Attachment_J.16_-_OST_Test_and_Evaluation_Guide.pdf

Office of Security Capabilities Test & Evaluation Process Guide

Version 1.0

May 22, 2013

Version 1.0 1

Table of Contents

1.0 INTRODUCTION

1.1 TSA T&E M ISSION

1.2 T&E OVERVIEW AND OBJECTIVES OF THE T&E PROCESS GUIDE

2.0 DHS ACQUISITIONS PROCESS

2.1 OVERVIEW OF THE ACQUISITION PROCESS

3.0 ENTERING THE TSA TEST & EVALUATION PROCESS

3.1 TEST & EVALUATION (T&E) STRATEGY OVERVIEW

3.1.1 Test & Evaluation Phases and Objectives

3.2 TSA STAKEHOLDER ROLES AND RESPONSIBILITIES

3.3 CREATION OF THIRD PARTY TESTING AND READINESS ASSISTANCE

3.3.1 Third Party Testing Processes and Levels of Stakeholder Engagement

3.3.2 Benefits of Third Party Testing

3.4 STREAMLINING OF DETECTION DEVELOPMENTAL TESTING

3.4.1 Readiness Assistance

3.4.2 Readiness Testing

4.0 DEVELOPMENTAL AND QUALIFICATION TESTING

4.1 TRANSITIONING FROM NON-TSA M ANAGED TESTING TO DEVELOPMENTAL TESTING

4.1.1 Developmental Testing Overview and Objectives

4.2 DEVELOPMENTAL TESTING STAKEHOLDER ROLES AND RESPONSIBILIT IES

4.3 DEVELOPMENT TESTING EVALUATION AREAS

4.3.1 Developmental Testing

4.3.2 Qualification Testing

4.3.3 Regression Test

4.3.4 Special Emphasis Tests (Customer Tests)

4.4 DATA AUTHENTICATION PROCESS

4.5 DEVELOPMENTAL / QUALIFICATION EXIT CRITERIA AND KEYS TO SUCCESS

5.0 OPERATIONAL TESTING

5.1 TRANSITIONING FROM DEVELOPMENTAL TESTING TO OPERATIONAL TESTING (OT)

5.2 OPERATIONAL TESTING STAKEHOLDER ROLES AND RESPONSIBILITIES

5.3 OPERATIONAL TESTING SITE SELECTION PROCESS

5.4 OPERATIONAL TESTING PHASES

5.4.1 Coordination

Version 1.0 2

5.4.2 Burn-in

5.4.3 Record Testing

5.4.4 Test Closeout and Site Restoration

5.5 DATA AUTHENTICATION PROCESS

5.6 OPERATIONAL TESTING EXIT CRITERIA

6.0 EVALUATION & QUALITY ASSURANCE

6.1 OVERVIEW OF THE EVALUATION PROCESS

6.1.1 Evaluating Effectiveness and Suitability

6.1.2 Evaluation Planning and the System Evaluation Plan (SEP)

6.1.3 Operational Test Reporting and the System Evaluation Report (SER)

6.2 OEM ROLE AND COMMUNICATION DURING THE EVALUATION PROCESS

6.3 UNDERSTANDING EVALUATION RESULTS AND NEXT STEPS

7.0 ACCEPTANCE TESTING

7.1 OVERVIEW OF ACCEPTANCE TESTING PROCESS AND OBJECTIVES

7.2 FIRST ARTICLE TESTING

7.3 FACTORY ACCEPTANCE TESTING

7.4 SITE ACCEPTANCE TESTING

7.5 INTEGRATED SITE ACCEPTANCE TESTING (ISAT)

8.0 TEST & EVALUATION MATRIX

8.1 SUMMARY OF T&E PHASES AND OBJECTIVES

9.0 OEM COMMUNICATION AND TEST & EVALUATION MILESTONES

9.1 OEM COMMUNICATION

9.2 M ILESTONES THROUGHOUT THE T&E PROCESS

APPENDIX A - GLOSSARY

Version 1.0 3

1.0 INTRODUCTION

1.1 TSA T&E MISSION

The mission of TSA Test & Evaluation (T&E) is to provide timely, accurate information to decision makers while reducing programmatic, financial, schedule, and performance risks associated with the development, procurement, deployment, and operation of security systems. T&E processes are intended to support three core TSA goals: improve operational performance and security effectiveness;

integrate new technologies into ongoing security operations; and facilitate competition among qualified manufacturers.

1.2 T&E OVERVIEW AND OBJECTIVES OF THE T&E PROCESS GUIDE

Former Secretary of Defense William Perry once described T&E as being the conscience of acquisition.

While the phrase has a certain poetic truth, it also carries negative connotations; a conscience is often relied upon to identify and maintain a straight and narrow path. It often asserts itself just when things begin to be fun or exciting; consequently, T&E may be viewed as an obstacle to an otherwise positive enterprise. T&E is often seen as the bearer of bad news or the illuminator of the reality that the system upon which extensive time and resources have been invested does not perform as well as the program manager or developer intended.

Regardless of one’s viewpoint, T&E is an essential component of the Office of Security Capability’s (OSC) security technology system development, acquisition, deployment, and sustainment mission responsibilities within TSA. Test and Evaluation are separate and distinct, yet complimentary processes that together provide timely and accurate information so that technology acquisition decision makers, users, and developers can make informed decisions regarding systems intended to meet TSA operational needs.

Testing is any program or procedure that is designed to obtain, verify or provide quantifiable data for the evaluation of research and development; progress in accomplishing development objectives; or functional, performance, and operational capability of systems, subsystems, components and equipment items. Testing is typically empirical and highly quantitative.

Evaluation is an independent process by which evaluators assess data from all available sources to determine if a system satisfies the approved requirements, and then document and report on that assessment in a way that decision authorities understand and find useful for decision making. Evaluation provides meaning to empirical data.

This Process Guide delineates the current state of T&E and seeks to share with TSA stakeholders a future vision of improvements to core T&E processes. In so doing, TSA intends to enlist the support of its stakeholders in meeting its acquisition goals. Results notwithstanding, T&E processes should never come as a surprise to either the sponsors of the test or the developers, nor should such processes be a convoluted series of unknown hurdles, disassociated from well-articulated user needs and requirements. The T&E Process Guide is intended to inform the full range of T&E stakeholders;

however, a primary audience is the Transportation Security Equipment (TSE) developer community. This includes existing as well as prospective Original Equipment Manufacturers (OEM), large or small, Version 1.0 4 desiring to do business with TSA to supply systems intended for acquisition by TSA.

This is a living document and as T&E processes evolve, so will this document. TSA will continue to focus efforts to introduce new security technology and capabilities to the marketplace more quickly, and to ensure that stakeholders are engaged and clearly understand our processes. The T&E portion of the overall acquisition process (discussed in Chapter 2) must become more efficient, and TSA will continue to improve the T&E processes to meet that goal. In the current fiscally constrained environment , TSA can no longer focus testing resources on equipment with an inadequate level of technical maturity. DHS S&T will continue to provide technology readiness assistance to support TSA and to help vendors in advancing technological maturity levels.

Version 1.0 5

2.0 DHS ACQUISITIONS PROCESS

2.1 OVERVIEW OF THE ACQUISITION PROCESS

TSA acquisitions are guided by DHS Acquisition Directive 102-01, which includes Test and Evaluation as an integral component of the Acquisition Life Cycle. The results from both Developmental Testing (DT) and Operational Testing & Evaluation (OT&E) activities drive acquisition decisions as well as the system’s progress through both the System Engineering Life Cycle (SELC) and ultimately the DHS Acquisition Life Cycle.

The associated instruction in Acquisition Directive 102-01 provides a flexible Acquisition Life Cycle Framework (ALF) for translating mission needs and gaps into cost-effective operational capabilities via stable and well defined acquisition events. The framework is designed to ensure that Program Management has the tools, resources, and flexibility to execute the acquisition, deliver a product tha t meets the user’s requirements, and complies with applicable statutes, regulations, as well as policies.

The overall framework is shown in Figure 2.1, “The DHS Acquisition Life Cycle,” for capital assets (IT and non-IT) and services.

Figure 2.1: The DHS Acquisition Life Cycle

The ALF includes links with the Department’s strategic requirements process as well as the Planning, Program, Budget and Execution, and supporting acquisition processes, such as test and evaluation.

The Acquisition Review Process (ARP) has two types of reviews. These are: 1) Acquisition Decision Events (ADE) used by the Acquisition Decision Authority (ADA) to assess program/project maturity and risk at pre-defined points in the acquisition life cycle, and 2) reviews defined in the Systems Engineering Life http://www.dhs.gov/xlibrary/assets/foia/mgmt_directive_102-01_acquisition_management_directive.pdf

Version 1.0 6

Cycle (SELC) Guide used by Program Management and the development authority to assess technical progress at pre-defined points along the SELC. SELC reviews are the basis for the ADE reviews according to the relationship shown in Figure 2.2.

Further, the DHS acquisition life cycle process is structured to operate within a series of acquisition phases, each leading to an ADE. This phased systematic approach to acquisition is a proven government and industry method for reducing acquisition risk and achieving more effective and efficient results from invested resources. The ultimate utility for Program Management and the operational end-users is better constructed acquisitions and more informed acquisition decisions. These, in turn, lea d to predictable and effective delivery of DHS capabilities.

Figure 2.2: Relationships between the Capital Planning and Investment Control (CPIC), Acquisition Life Cycle Process and Systems Engineering Life Cycle (SELC) Phases

To support DHS acquisition strategy, the SELC provides a framework for development using proven systems engineering principles and is tailored to fit the unique circumstances of the program/project.

SELC reviews are used to inform the Component/Department oversight structure (e.g., ADE reviews) on the technical progress towards successful capability development. The activities associated with completing each phase of the SELC vary for each type of system solution. Details related to the requirements associated with each phase are provided in the solicitation materials.

The SELC framework accommodates three types of system solutions:

Commercially Off The Shelf (COTS) – System baseline is a baseline configuration that has been utilized and commercially available for a minimum of 3 Months. All Configuration Items (CI) are Non-Developmental Items.

Version 1.0 7

Modified COTS – System baseline is a baseline configuration that have has been utilized and commercially available for a minimum of 3 Months but include CIs that have been refined or newly developed.

Developmental – System baseline is newly developed.

Each type of system solution is subject to varying degrees of requirement/specification compliance test and evaluation. To fully evaluate a system under test (SUT)’s performance, both developmental and operational testing occurs. The TSA primarily conducts developmental testing efforts to answer the question “Does the system perform as specified or required?” Conversely, the TSA conducts operational testing to answer “Does the system work in the intended environment?” Vendors must consider the system’s overarching mission when designing potential systems. This is accomplished through analysis of the Operational Requirements Document (ORD), which outlines mission requirements and constraints. The Concept of Operations (CONOPS) document provides background on how a system or capability will be employed and supported in the field and provides input into developing test scenarios.

The system design constraints are communicated through the functional and performance requirements in the Functional Requirements Document (FRD). To increase the likelihood of successful system delivery (i.e., a positive ADE 3 decision), both the functional and operational requirements constraints must be addressed.

Test & Evaluation’s role within the ALF is to provide the ADA and other stakeholders with unbiased information regarding the proposed system’s capabilities and limitations. DHS and TSA clarify OSC T&E’s responsibilities under the AD-102 construct within the DHS Management Directive 026-06 and TSA MD

300.21. These activities occur primarily in the ALF’s “Obtain” phase, between ADE 2B and ADE 3. This is accomplished through a determination of system effectiveness (i.e. , does the system work?) and suitability (i.e., is the system practical and can we count on it?). The various test organizations provide developmental and operational test results through event reports that culminate in a System Evaluation Report (SER). The SER provides effectiveness and suitability determinations, along with potential system improvement recommendations based on information collected during both qualification and operational testing (the SER process is discussed in more detail in Section 6 Evaluation & Quality Assurance).

The decision to purchase a proposed system occurs at ADE 3. To achieve a successful ADE 3 outcome, the ADA must consider cost and benefits information, as well as Test and Evaluation results (among other items).

Key to the overall test strategy is the Test and Evaluation Master Plan (TEMP). The TEMP is the “top-level” planning document for all T&E related to a particular capital asset program or project. Its primary purpose is to describe the program’s T&E strategy in terms of the Developmental and Operational testing needed to determine system technical performance, and the strategy for evaluating the system’s operational effectiveness and suitability through an integrated assessment of that developmental and operational testing.

Version 1.0 8

3.0 ENTERING THE TSA TEST & EVALUATION PROCESS

3.1 TEST & EVALUATION (T&E) STRATEGY OVERVIEW

TSA is refining its T&E strategy in order to accomplish two objectives. First, it must verify satisfaction of the performance requirements associated with a system that TSA requires to meet a mission need.

These requirements are often described in terms of technical specifications in the FRD. Second, it must answer the questions raised by the critical operational issues that drive the requirements for a given security system. Such requirements and operational issues may be allocated to developmental tests, operational tests, factory acceptance tests, or site acceptance tests.

In order to acquire new and innovative security systems that can be optimally integrated into TSA security operations while meeting essential mission requirements, qualification testing is intended to:

Verify the attainment of performance specifications and operational requirements for systems and subsystems under consideration

Verify that integrated systems and their component subsystems are operationally effective and suitable for their intended use in the field.

Provide essential information for assessing technical and acquisition risks associated with a system

Provide essential information to support programmatic decision-making and align with the structured DHS investment review processes described in Section 2 of the Process Guide

Test objectives are intended to mitigate potential operational risks and to demonstrate system performance. Quantitative criteria are used for the analysis of hardware, software, as well as system maturity and readiness to proceed through the acquisition management process.

3.1.1 Test & Evaluation Phases and Objectives

To achieve the overarching goal of effective system procurement and deployment, TSA employs a series of rigorous testing phases to fully qualify a system. The T&E process can be divided into three major components:

Developmental Testing (DT), including formal Qualification/Certification Testing (QT)

Operational Testing (OT), which occurs prior to and in support of procurement decisions

Factory Acceptance Testing (FAT) and Site Acceptance Testing (SAT), which combined are referred to as Acceptance Testing (AT).

The purpose of each phase of testing is to both verify and assess different aspects of system design, effectiveness, as well as suitability, and is summarized in Figure 3-1.

Operational effectiveness and operational suitability are terms that have great significance with regard to the qualification of systems for procurement and deployment in the field. Effectiveness refers to the degree to which a product accomplishes its mission when used by representative personnel in the expected operational environment. Effectiveness testing may include probabilities of detection, false

Version 1.0 9 alarm rates, throughput, degraded mode operations, as well as safety and security issues. Suitability refers to the degree to which a product intended for field use satisfies its availability, compatibility, transportability, interoperability, reliability, maintainability, safety, human factors, logistics supportability, and training requirements.

Table 3-1. T&E Testing Objectives

Test Objective DT / QT OT AT

Verify contractor compliance to contract functional and performance requirements.

Identify deficiencies in system design and documentation, hardware, software, human performance factors, or operational concepts.

Identify and demonstrate mitigation of risks.

Resolve critical operational issues.

Assess operational effectiveness, supportability (including maintenance), and suitability, including the human component.

Verify that the system meets reliability, maintainability, and availability requirements.

Evaluate the compatibility and interoperability with existing or planned systems or equipment.

Assess system operations in a degraded mode.

Verify that the system is safe, secure, and survivable.

Assess the site adaptability of the system.

Assess the transition switchover capability/plan.

Verify the adequacy of manuals, handbooks, supporting plans, and other documentation for operations, maintenance, and training.

Assess the degree to which the system can be operated and maintained by users in an operational environment.

Verify system operations under stress and loading.

Assess end-to-end performance with the system installed to ensure that pre-existing system functionality is not degraded by new system insertion/integration.

Ensure production units are of consistent quality and are equivalent to the first article.

Verify that production units are free from manufacturing defects.

Verify that installation and integration of fielded systems is consistent with approved SAT plans.

*Note that any single physical test may address multiple developmental, operational, or other objectives.

DT can be traditionally divided into two aspects: exploratory and confirmatory. Exploratory DT provides feedback to system developers in order to mature the technology. The primary objective of confirmatory DT is to verify that all technical and performance requirements specified in the contract, solicitation, and/or specification have been met. DT concludes with a formal qualification test that verifies the system’s ability to satisfy the system specification in support of an acquisition decision.

Historically, the DHS Science & Technology (S&T) Transportation Security Laboratory (TSL) and the TSA Systems Integration Facility (TSIF) have been primarily used to perform QT. In addition, the TSIF “bridges” the gap between DT and OT, with elements of both types of testing being routinely conducted.

The primary objective of Operational Testing (OT) is to validate that a new or modified system is operationally effective and operationally suitable for use by TSA, and that the infrastructure is ready to

Version 1.0 10 accept the system. These tests focus on verifying that operational requirements are met and that all Critical Operational Issues (COIs) are resolved. Operational testing is conducted by the TSA at field sites using field-representative personnel with production-representative systems.

After completion of OT, and prior to the ALF Produce/Deploy/Support phase, an Acquisition Decision Event (ADE) at ADE-3 must be made to support proceeding to full-rate production of the system. Upon procurement award, the OEM will produce the first article unit for First Article Test and Evaluation (FAT&E). FAT&E will be performed by the OEM and TSA representatives at the OEM manufacturing facility and TSA facilities, as necessary, to verify the produced system meets all applicable specification requirements and performance characteristics. After the first article of a new system is accepted, reproductions of that system can proceed upon ordering by TSA. Factory Acceptance Testing (FAT) is conducted by the vendor and witnessed by the government on each replicated system before it leaves the factory. Factory tests usually are a subset of the qualification tests and verify the consistency of the OEM production and manufacturing process.

Following delivery to the site, Site Acceptance Testing (SAT) is performed. These tests are conducted by the vendor and witnessed by the government to ensure that the system is installed and functioning properly. For complex integrated systems, such as in-line Explosive Detection Systems (EDS) that are installed within checked baggage inspection systems (CBIS), an integrated SAT (ISAT) is performed by government representatives and the Integrated Local Design Team (ILDT) to ensure the integrated system performs to its specifications and requirements.

3.2 TSA STAKEHOLDER ROLES AND RESPONSIBILITIES

Throughout the T&E process, OEMs may interface with a multitude of external and internal TSA stakeholders. External stakeholders include the traveling public, air carriers, airport operators, as well as system manufacturers and integration contractors. Internal stakeholders include the Office of Security Operations (OSO), the organization that oversees Federal Security Directors (FSD) and Transportation Security Officers (TSO), who are responsible for security operations in the field. OSO sponsors systems acquisition programs managed by the Office of Security Capabilities (OSC). OSO and OSC establish the performance baseline and operational framework that underpins the operational requirements of systems. OSO represents the user community and OSC houses the program management offices that are responsible for fielding operationally suitable and effective systems for checkpoint, checked baggage, and air cargo screening. Although there are many stakeholders, OSC serves as the primary point of contact for development, acquisition, deployment, and life-cycle support of TSA’s security capabilities.

The TSL and other DHS S&T entities are component organizations that provide testing and technical expertise to the T&E organization in planning and conducting system test activities. These activities include preparation of required test documentation and test articles, coordination and conduct of system tests, and reporting the results of these tests to the TSA system evaluator. In addition, S&T provides DT facilities and guidance to system manufacturers seeking to place their products on the TSA Qualified Products List (QPL).

The Director of Operational Test & Evaluation (DOT&E) is housed within DHS S&T and is responsible for oversight of all operational testing activities as well as the development and implementation of DHS-wide T&E policies and procedures. DOT&E is charged with approving Test and Evaluation Master Plans

Version 1.0 11

(TEMP) and Operational Test Plans (OTP) that describe the necessary DT and OT tasks that must be conducted in order to determine system technical performance and operational effectiveness based upon vetted operational requirements. DOT&E approval of T&E plans and documentation is required for the department’s acquisition review process. DOT&E is provided test plans, reports, and observes testing to maintain knowledge of developmental and operational testing.

3.3 CREATION OF THIRD PARTY TESTING AND READINESS ASSISTANCE

As a function of T&E strategy refinement, TSA and S&T are looking to the system manufacturers to assist in the development of additional testing capabilities and readiness assistance support. The purpose of refining the strategy of OEM readiness assistance is to expedite the acquisition and deployment of suitable and effective systems for use in support of TSA security operations.

As a result, TSA supports the establishment of preliminary system development gateways by identifying and, in the future, qualifying capable third party testing facilities. The purpose of creating this initial step is to assist OEMs in developing more mature systems prior to entering the formal TSA T&E process.

Thus, this form of readiness assistance will allow OEMs to more efficiently proceed through the testing process because systems will be more capable of meeting TSA operational effectiveness and suitability requirements. This will also allow TSA to apply its limited T&E resources more expeditiously for systems demonstrating a mature level of readiness for operational testing.

Under current acquisition guidelines and operational test policies that are derived from Department of Defense (DoD) test management regulations, TSA must limit OEM involvement during and directly after the conduct of government testing. This limits the communication between the OEM and the government test agents and, as a result, delays the OEM’s mitigation of observed deficiencies. Third party testing is intended to assist OEMs by allowing open communication between the third party tester and the OEM, with government assistance as available, which will facilitate faster development of OEM resolutions to identified deficiencies. In addition, third party testing facilities with baggage handling systems commissioned to meet the critical security and tracking requirements identified in the Planning Guidelines and Design Standards (PGDS) will greatly assist OEMs in identifying integration issues commonly seen with developed EDS systems. While not exclusively intended for EDS, these preparation activities would significantly reduce the cycle time required for meeting TSA-managed OT within airport venues.

Third party testing will also further TSA/OSC efforts to deliver effective and innovative security technologies. TSA has consistently embraced three core goals related to its acquisition of new systems and technologies:

Deliver systems demonstrating improved operational performance and security effectiveness

Accommodate successful integration of new technologies with ongoing security operations and existing technologies and infrastructure

Facilitate competition among multiple qualified manufacturers

TSA believes that encouraging third party testing and providing OEMs with functional and operational requirements specific to systems under development, OEMs will submit more mature systems to TSA for qualification testing. Increased system maturity will assist TSA in reducing acquisition timelines by

Version 1.0 12 curtailing the need for the multiple and iterative test activities that are currently applied when systems lack the required level of maturity for operational testing.

3.3.1 Third Party Testing Processes and Levels of Stakeholder Engagement

As a function of the third party facility identification process, TSA will ensure that third party test entities demonstrate core capabilities in terms of handling classified and Sensitive Security Information (SSI). In addition, TSA will consult with the test agents to review/recommend proposed detailed test procedures intended to assess capability against operational and functional requirements. Outside of the conduct of a test event, OEM communication with the test agent will not be restricted. In this manner, deficiencies uncovered by test events can be rapidly addressed. TSA test personnel, if available, may advise the third party test agents on test procedures, but will not be directly involved in testing events.

3.3.2 Benefits of Third Party Testing

Testing conducted by a third party facility can provide benefits to both the OEM and, ultimately, to TSA.

As mentioned, the OEM will benefit from the rapid engineering development opportunities that will result from the testing in a collaborative environment unrestricted by federal procurement sensitivities.

Since engineering changes can be quickly evaluated at the test facility, OEMs will not incur repetitive shipping costs between test events. Additional benefits will accrue from the review and feedback on required OEM user and maintenance manuals and documentation. For EDS OEMs, the exposure to a variety of baggage handling system components and control interfaces will provide valuable information relative to optimum integration configurations.

As the third party testing concept matures and evolves, TSA would look to sponsor certain test events, making use of the information gained in an effort to reduce the overall cycle time of TSA required test events. The periodic use of a third party test facility for government sponsored testing would in no way impact the collaborative relationship between the test agent and the OEM for testing outside of government sponsorship.

Testing will also significantly reduce TSA procurement delays associated with adverse operational test events in integrated environments. The expertise of the OEMs is primarily associated with the knowledge of their hardware, as opposed to its interface in a CBIS. The third party testing conducted in an in-line operational test matrix will take the view of the systems under test not as components, but as part of a system mimicking a deployed airport environment. Such testing will provide the OEM the freedom to spend as much time as they need to collect performance data and incorporate lessons learned in their system development. Ultimately, this testing is a preliminary effort to streamline residence at the TSIF and to predict performance in the field, which will minimize the drain on TSA and OEM resources by providing more efficient feedback loops.

3.4 STREAMLINING OF DETECTION DEVELOPMENTAL TESTING

In partnership with TSA, S&T is seeking to develop advanced information analysis tools and test objects to enhance OEM in-house detection system development and testing capabilities. A notional schedule for the development of these information analysis tools and test objects is shown in Figure 3-2 below.

The primary goal of these efforts is to reduce overall developmental testing resource investment and

Version 1.0 13 time to market. The development of standardized test objects for OEM and Government use would also support third party testing initiatives for TSA.

Figure 3-2: Notional Schedule for the Development of Advanced Data Analysis Tools and Test Objects

The future availability of improved data analysis capabilities and test sets for OEM use is expected to allow system manufacturers to be better prepared for developmental testing by conducting more in-house data collection that has traditionally been conducted by the Government. The results of this in-house testing would then be reported as part of the Qualified Data Package (QDP) and validated at a Government facility as deemed appropriate by the DHS. Developmental data collection and testing may still be conducted by the Department through Readiness Assistance and Readiness Testing. Initiation of projects for DT may require a formal agreement, such as a bailment or cooperative research and development agreement (CRDA), in order for the system to be tested by the Government. S&T also directly supports developmental testing as part of their RDT&E program efforts in support of TSA requirements.

3.4.1 Readiness Assistance

Readiness Assistance (RA) is intended for OEMs providing prototype or production ready systems requiring T&E data and engineering development guidance. RA may be in conjunction with a particular TSA acquisitions window but that is not a requirement. RA may also be useful for OEMs intending to

Summary Schedule (24 months)

Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8

GFE for Task Areas:

X-ray test bed prototype (CAXI or other)

GFI for Task Areas:

Signature data, airport stream-of-commerce bags

Task Area 1: X-ray Test Bed

Prototypes

Task Area 2: Supporting

Analytical Tasks

Information Theoretic Analysis

Classification on Data Sets

Automated Decision Aids

Priors Library

Task Area 3: T&E Support

EDS/AT Assessment

Test Articles

Task Area 4: Architectural

Components

Sources & Detectors

Task Area 5: X-ray System

Architectural Design Concepts

Ref: Industry Days

PDR CDR

Test Window 1

Test Window 2

SMR+DR Option: prototype experiments2

PDR CDR

SMR

CDRPDR

classification w/vendors

SMR

CDRPDR

test cases

SMR

CDRPDR

SMR

CDRPDR

build, T&E, delivery

Generation 1 SMR

PDR

build, T&E, delivery

CDR

SCR

SMR

PDR1

Trade Study 1

Final Trade Study

PDR2

From X-ray Test Bed Prototype (with new signature measurement technology)

Prototype build, test (option)

SCR

(from EDS & AT platforms)

SCR SMR

SMR

SCR

Interim Tech Analysis Report4

Option: prototype experiments2

SMR: Signature Metrics Review

SMR

SMR

SMR data collection at TAFB1 GFE: available for prototype experiments2 plan3

SMR

BAA

GFE

GFI

PDR CDR

build, T&E, delivery plan3Near COTs

Non-COTs

SCR

Version 1.0 14 modify existing systems. RA may define a spiral development approach to support technology maturation to include baseline tests and up to three regressions.

3.4.2 Readiness Testing

Readiness Testing (RT) is intended for OEMs involved in a certification or qualification window. RT will involve evaluation of the system’s detection sensitivity determined at or near the threat level specified for the qualification or certification requirements. Readiness Testing will be prioritized based on TSA acquisition needs and schedule.

Version 1.0 15

4.0 DEVELOPMENTAL AND QUALIFICATION TESTING

4.1 TRANSITIONING FROM NON-TSA MANAGED TESTING TO DEVELOPMENTAL TESTING

Upon completion of third party testing or in-house development, OEMs will be required to submit a detailed Qualification Data Package (QDP) providing evidence of compliance with the contract and FRD.

The primary objectives of the QDP are to convey the information needed by the government to:

Ensure that the OEM has provided all substantiation required

Verify that the OEM’s data substantiates that the system functionality is likely to meet the requirements

Verify that the OEM has conducted its own system assessments using a production or production-representative system configuration. The system used to develop substantiation must be production-representative and not specifically developed, selected, or tuned for qualification test purposes.

OEMs will be required to submit a QDP to include all material specified in the Qualification Management Plan (QMP) for the system under contract. Typically QMP are provided as part of a technology development contract between the OEM and TSA or DHS S&T. This includes specific identification of the system baseline tested by the OEM, a test plan detailing the approach to be used during the OEM and independent third party data collection, test data, and analysis results. The QDP must include all information and material necessary to confirm that the system meets the specifications. In addition, complete identification of the system used to collect the substantiation data must also be provided in the QDP. This includes:

Model number(s) and serial number(s) of equipment tested

Software version(s) and firmware version(s), if applicable

An inventory of hardware Configuration Items (CIs), software CIs and firmware CIs with configuration identification data

Identification of all configurable and adjustable parameters and adaptation data used to tailor the system, as well as their settings and values

Going forward, TSA will be refining the QDP process to incorporate third party testing and to enhance the level of information included in the QDP submission. This will allow TSA to better assess system capabilities to enter DT, verify the results of third party or in-house testing, as well as reduce the likelihood and frequency of system failures during formal testing.

4.1.1 Developmental Testing Overview and Objectives

The DT program supports the acquisition strategy and the systems engineering process, providing the information necessary for informed decision making throughout the development process and at each acquisition milestone. DT involves the verification and validation of the systems engineering process and must provide confidence that the system requirements meet the desired functional and performance

Version 1.0 16 capabilities. DT generally begins with component and sub-system testing and then increases to system level assessments. Robust DT reduces technical risk and increases the probability of a successful OT.

Objectives of DT include:

Perform verification the system meets the requirements of the FRD

Verify the limits of the system (as defined by the CONOPS/SOP) to test the performance and capabilities and ensure that expected operational performance environments can be satisfied

Assess the safety of the system to reduce injury to typical users during normal operations

Provide data and analytic support to the decision process to certify the system is ready for OT

Identify technological capabilities and limitations of alternative concepts and design options under consideration to support cost-performance tradeoffs

4.2 DEVELOPMENTAL TESTING STAKEHOLDER ROLES AND RESPONSIBILITIES

The two primary stakeholders and performers of developmental / qualification testing are the DHS S&T Transportation Security Laboratory (TSL) and the TSA Systems Integration Facility (TSIF).

Role of the DHS S&T Transportation Security Laboratory (TSL)

The TSL is typically responsible for Qualification Testing (QT), which includes Explosive Detection System (EDS) and Explosive Trace Detection (ETD) Certification Testing (CT). This program is primarily reserved for detection performance testing and is focused on certifying that salient performance characteristics, such as the probability of detection of all categories of explosives with appropriate false alarm rates and throughput rates, are met.

Role of the TSA Systems Integration Facility (TSIF)

The TSIF enables the TSA to evaluate transportation security systems, support infrastructure, and mission profiles in a controlled environment. The TSIF test program is primarily DT/QT oriented addressing conformance with technical requirements under controlled conditions in laboratory or a simulated operational environment. Laboratory space includes a Baggage Handling System (BHS) as well as infrastructure and space to host stand-alone EDS, a passenger screening laboratory with multiple system layouts, a floor area with the capacity and infrastructure to test cargo and vehicle screening systems, and a laboratory dedicated to ETD and Bottle Liquid Scanner (BLS) system testing. TSIF also bridges the gap between the DT/QT environment and the OT conducted by the Operational Test Section (OTS) by involving end users such as Transportation Security Officers (TSO) to exercise standard operating procedures and provide feedback from a human interface perspective.

4.3 DEVELOPMENT TESTING EVALUATION AREAS

4.3.1 Developmental Testing

Development Testing (DT) verifies and validates the systems engineering process by providing confidence that the system solution will satisfy sought after functional and performance capabilities.

The purpose of DT is to advise TSA on technical progress for a developmental system and to confirm

Version 1.0 17 compliance with requirements. DT conducted prior to qualification testing described below is typically “exploratory” and provides data and analytic support to determine system readiness to enter the formal TSA testing process.

4.3.2 Qualification Testing

Qualification Testing (QT) is “confirmatory” developmental testing and is the formal verification and validation of system performance and must ensure that the system satisfies functional requirements. QT typically supports an acquisition decision and provides confirmation that the system complies with the criteria outlined in the procurement strategy and with requirements defined in the Functional Requirements Document (FRD). Similar to DT, QT also provides data and analytic support to determine that the system is ready to enter operational testing.

4.3.3 Regression Test

Regression testing verifies implementation of system changes to satisfy requirements and does not negatively impact performance in unintended areas. The purpose of regression testing is to assess configuration changes or to determine adequacy for operational fleet fielding.

4.3.4 Special Emphasis Tests (Customer Tests)

Customer tests are conducted as needed to assess and/or compare system integration/interoperability, Reliability, Maintainability, and Availability (RMA), Human Factors, and any other area of interest.

Typical customer tests are described in the sections below.

4.3.4.1 Comparison Tests

Comparison tests assess various procedures, system settings, or other parameters against a baseline for comparable systems.

4.3.4.2 Integration Test / Interoperability

Integration/Interoperability testing assesses system performance when integrated into typical infrastructure including associated information technology. Integration testing verifies interoperability with other equipment that may be deployed with the system under test. These tests also verify interoperability/integration with typical infrastructure such as baggage handling systems. It also evaluates systems installation and integration procedures and interoperability within installed networks.

Integration testing also evaluates the system’s interoperability within a network and evaluates the following elements for suitability within a system:

Interoperability

Transportation Package and handling

Documentation (installation procedures) Maintainability

Security Technology Integrated Program (STIP)

Version 1.0 18

Interoperability testing evaluates a system’s performance within the context of other systems and term is often interchangeable with integration testing.

4.3.4.3 Reliability, Maintainability and Availability Test (RMA)

RMA testing assesses system mission performance based on operational duty cycles or mimicking duty cycles similar to those expected in an operational environment. It provides information on whether a system can perform the mission for a specified time, be retained in or restored when maintenance is accomplished, and is available for operations. RMA testing confirms compliance with requirements and tests system stability. RMA may also be done in support of an acquisition decision. In addition, it can be used to assess field forensics and inform risk reduction for fielding.

4.3.4.4 Human Factors Test

Human Factors testing is a broad concept that encompasses a wide variety of specific concepts that center on the human-machine interface, training, and other requirements. It’s a type of special emphasis testing focused on human-system interfaces and is typically done to determine adequacy for fielding.

4.3.4.5 Threat Mitigation Support

Threat mitigation support is used on an ad-hoc basis to address new threats that occur in the field and to test current technology on its ability to detect specified threats. It also supports the assessment of system responses for various threats and concealment methods.

4.4 DATA AUTHENTICATION PROCESS

A Data Authentication Group (DAG) consisting of test personnel, PMO representatives, user representatives, and subject matter experts serves to authenticate and validate test data. Chaired by the government test lead, this group meets periodically to ensure the data captured as part of the test process is complete, accurate, consistent, and representative of the test activities.

4.5 DEVELOPMENTAL / QUALIFICATION EXIT CRITERIA AND KEYS TO SUCCESS

Upon completion of QT, any non-compliance issues are addressed with actions assigned for resolution of each item. Non-compliance issues are scored and assessed as a part of an overall determination of readiness of the system to move to OT&E.

In addition to the system under test showing demonstrated capability to satisfy all Functional and Performance Requirements contained in the procurement documentation, OEMs can achieve success through development and application of robust training and preventative maintenance programs that consider the end-use environment under consideration.

Version 1.0 19

5.0 OPERATIONAL TESTING

5.1 TRANSITIONING FROM DEVELOPMENTAL TESTING TO OPERATIONAL TESTING (OT)

Transition activities to OT will begin once a SUT successfully meets DT exit criteria. The critical component of the exit criteria is a determination that the system has met established technical requirement performance and expected suitability results. Upon successful completion of QT, the vendor will provide the appropriate systems for shipment to a designated test site(s). The Systems Integration (SI) contractor will install and integrate the systems in the operational environment and perform Site Acceptance Testing (SAT)/integrated Site Acceptance testing (ISAT) to determine system readiness to proceed into the burn-in phase of OT. An Operational Test Readiness Review (OTRR) will be held to confirm readiness criteria to enter the burn-in period, and again to confirm readiness criteria to enter record testing. Test readiness reviews will be discussed in greater detail below.

5.2 OPERATIONAL TESTING STAKEHOLDER ROLES AND RESPONSIBILITIES

As in other testing phases, OT involves a variety of stakeholders across TSA and DHS.

Table 5-1: Operational Testing Stakeholder Responsibilities

Stakeholder Role / Responsibility

Operational Test Section (OTS) Oversees all aspects of the OT to include coordination and planning, burn-in, and record testing.

Evaluation and Quality Assurance (EQA) Section

Responsible for the overall evaluation of the SUT to include development of the System Evaluation Plan (SEP) and System Evaluation Report (SER).

Program Management Office (PMO) Responsible for overseeing all aspects of the system acquisition life cycle. In support of OT, the PMO establishes the acquisition strategy and budget, monitors OT progress against schedule, participates in Data Authentication Group (DAG) sessions, and serves as the liaison between TSA and the vendor.

Office of Security Operations (OSO) Represents the operational user community/OSC Deployment Division throughout OT activities. Responsibilities include providing inputs to T&E documentation, participating in DAG meetings, assisting with OT site selection and coordination, and development of system Concept of Operations (CONOPS) and Standard Operating Procedures (SOPs)

Deployment Division Responsibilities include providing inputs to T&E documentation, participating in DAG meetings, assisting with OT site selection and coordination, and development of system Concept of Operations (CONOPS) and Standard Operating Procedures (SOPs)

Version 1.0 20

Stakeholder Role / Responsibility

Department of Homeland Security (DHS) Science and Technology (S&T) DOT&E

In support of Level 1 acquisition efforts subject to DHS oversight only, provides approval for T&E artifacts to include the Test and Evaluation Master Plan (TEMP) and Operational Test Plan (OTP). Provides a Letter of Assessment (LOA) indicating DHS assessment of the conducted OT.

Additional Stakeholder Organizations There are a number of additional organizations that provide inputs to OT, including the Office of Occupational Safety, Health and Environment (OSHE), Office of Public Affairs (OPA), Office of Training and Workforce Engagement (OTWE), Acceptance Testing, and other DHS components.

5.3 OPERATIONAL TESTING SITE SELECTION PROCESS

During the test planning phase, the TSA System Evaluation Team (SET) develops test site criteria to facilitate a complete, unbiased evaluation of the SUT. Working together with cross-organizational TSA stakeholders, primary and alternate test sites will be selected that satisfy the site selection criteria. TSA T&E, Deployment, and OSO representatives will normally perform site visits to the candidate sites during the site selection process to determine system integration requirements and discuss the ability of the airport to support test requirements prior to finalizing the OT sites.

5.4 OPERATIONAL TESTING PHASES

OT entrance criteria must be satisfied prior to the system entering record testing. Entrance criteria include, but are not limited to, successful completion of DT/QT, approved maintenance concept, final SOPs, completed TSO training, and completed test planning efforts. All test entry criteria must be in approved status in order for a system to formally enter OT.

5.4.1 Coordination

Several coordination and preparation activities are required prior to the start of burn-in and record testing. TSA, in coordination with the OEM, will install and integrate the SUT in the operational environment. Usually, and especially in those instances where significant airport modification or integration into a baggage handling system is required, TSA will utilize the services of a systems integrator to work with the OEM to install the system at the test site. For simple installs where no airport modifications or integration is required, the OEM may be tasked to install the system in preparation for OT.

Once complete, either the TSA SAT or OTS teams will perform a SAT/ISAT to verify the system has been properly installed and integrated in the operational environment and verify the configuration baseline is consistent. During the coordination period, T&E, OSO, and other stakeholders, as required, will perform site visits to coordinate the test plan and schedule with the airport FSD management team and other parties as appropriate (i.e., airline representatives, law enforcement). In order to be considered “under test,” the OEM will be asked to supply TSA with an artifact stating their system is ready to test, regardless of whether the system is in burn-in or record test.

Version 1.0 21

5.4.2 Burn-in

Once the SUT is installed and integrated in the operational environment, a burn-in period will take place where TSOs will operate the system during live operations. An Operational Test Readiness Review (OTRR) is held prior to burn-in to ensure the system, site, and users are ready to enter into testing.

OTRRs are an internal T&E function and OEMs are not allowed to participate. The purpose of the burn-in period is for the system to stabilize in the operational environment,…