Attachment_J.15_-_OST_Test_and_Evaluation_Guidebook.pdf

Office of Security Technology Test and Evaluation

Guidebook

Version 1.0

August 21, 2009

Version 1.1 i

PREFACE

The purpose of this document is to establish a common framework for planning, coordinating, executing, and reporting independent and integrated test and evaluation activities for the Office of Security Technology (OST). Development of the common framework and vernacular will re-sult in improving the efficiency and effectiveness of OST integrated Test and Evaluation (T&E) planning, execution and reporting activities.

The Guidebook will be created in a modular, evolutionary manner. This Guidebook’s initial re-lease focuses on core T&E procedures and terminology to lay the foundation for a solid, repeata-ble, and independent OST Integrated T&E process. Periodically, the Integrated T&E Tiger Team will update this Guidebook to incorporate and address newly developed policies and procedures, acquisition regulations, and management directives. Additionally, future releases of this Guide-book will incorporate lessons learned and advanced T&E methodologies and concepts.

Version 1.1 ii

TABLE OF CONTENTS

SECTION NO. PAGE NO.

1.0 INTRODUCTION

1.1 TSA T&E MISSION

1.2 OBJECTIVES OF THE GUIDEBOOK

1.3 SCOPE

1.4 ROLE OF T&E FOR TSA ACQUISITION

2.0 ROLES AND RESPONSIBILITIES

2.1 INTEGRATED PRODUCT TEAM (IPT)

2.2 T&E WORKING-LEVEL INTEGRATED PRODUCT TEAM (T&E WIPT)

2.3 SYSTEM EVALUATION TEAM (SET)

2.4 IPT/T&E WIPT/SET RELATIONSHIP

2.5 PROGRAM MANAGEMENT OFFICE (PMO)

2.6 DHS S&T TRANSPORTATION SECURITY LABORATORY (TSL)

2.7 OST TSA SYSTEMS INTEGRATION FACILITY (TSIF)

2.8 OST EVALUATION & OPERATIONAL INTEGRATION (E&OI)

2.9 OFFICE OF SECURITY OPERATIONS (OSO)

2.10 OST ENGINEERING

2.11 OVERSIGHT AUTHORITIES

2.11.1 DHS S&T Test & Evaluation and Standards Division (TSD)

2.11.2 Acquisition and Program Management Support Division (APMSD)

2.12 OTHER ORGANIZATIONS

3.0 INTEGRATED TEST AND EVALUATION

3.1 OVERVIEW

3.2 EVALUATION STRATEGY DEVELOPMENT

3.2.1 Developing COIs

3.2.2 Evaluation Planning

3.2.3 Test and Evaluation Master Plan (TEMP)

3.2.4 System Evaluation Plan (SEP)

4.0 DEVELOPMENTAL TESTING (DT)

4.1 DT EVALUATION AREAS

4.1.1 Integration Testing

4.1.2 Interoperability Testing

4.1.3 Technical Conformance Testing

4.1.4 Reliability, Maintainability and Availability (RMA) Testing

4.1.5 Human Factors Testing

4.1.6 Acceptance Testing

4.2 DEVELOPMENTAL TEST READINESS REVIEW (DTRR)

4.2.1 Entrance Criteria

4.2.2 Exit Criteria

5.0 OPERATIONAL TESTING (OT)

5.1 OT TYPES OF TESTS

5.1.1 Initial Operational T&E (IOT&E)

5.1.2 Follow-on Operational T&E (FOT&E)

5.1.3 Limited User Test (LUT)

5.1.4 Integrated Site Acceptance Test (iSAT)

5.1.5 Customer Test

Version 1.1 iii

5.2 OPERATIONAL TEST READINESS REVIEWS (OTRR)

5.2.1 Entrance Criteria

5.2.2 Exit Criteria

5.3 SITE SELECTION

5.4 OT EXECUTION

5.4.1 Test Phases

5.4.2 Data Collection

5.4.3 Test Incident Reporting (TIR)

5.4.4 Data Authentication Group

5.4.5 Data Reduction and Analysis

5.5 OT REPORTING

5.5.1 Test Status Reporting

5.5.2 Emerging Results

5.5.3 Modeling & Simulation Results

5.5.4 Bottom Line Summary Brief

5.5.5 System Assessment (SA)

5.5.6 System Evaluation Report (SER)

APPENDIX A. ACRONYM DEFINITIONS ...................................................................................................................... A-1 APPENDIX B. GLOSSARY ............................................................................................................................................. B-1 APPENDIX C. COMMON DATA ELEMENTS ................................................................................................................ C-1 APPENDIX D. BASELINE CORRELATION MATRIX (BCM) SAMPLE ....................................................................... D-1 APPENDIX E. DATA SOURCE MATRIX (DSM) SAMPLE ............................................................................................ E-1 APPENDIX F. REQUIREMENTS CROSSWALK MATRIX (RCM) SAMPLE ................................................................ F-1 APPENDIX G. VERIFICATION REQUIREMENTS TRACEABILITY MATRIX (VRTM) SAMPLE .............................. G-1 APPENDIX H. SYSTEM EVALUATION PLAN (SEP) TEMPLATE .............................................................................. H-1 APPENDIX I. SYSTEM EVALUATION REPORT (SER) TEMPLATE ............................................................................ I-1 APPENDIX J. SYSTEM ASSESSMENT TEMPLATE .................................................................................................... J-1 APPENDIX K. EMERGING RESULTS BRIEFING (ERB) TEMPLATE ........................................................................ K-1 APPENDIX L. LETTER TEST AND EVALUATION PLAN (LTEP) ................................................................................ L-1 APPENDIX M. TEST EVENT PLAN (TEP) ..................................................................................................................... M-1 APPENDIX N. TEST EVENT REPORT (TER) ................................................................................................................ N-1 APPENDIX O. TEST STATUS REPORT (TSR) ............................................................................................................. O-1 APPENDIX P. T&E RESPONSIBILITY AND ASSIGNMENT MATRIX ......................................................................... P-1

Version 1.1 iv

LIST OF FIGURES

FIGURE NO. PAGE NO.

Figure 1. T&E Phases within the Acquisition Life Cycle Figure 2. TEMP and SEP Development Process Figure 3. Common Evaluation Areas Figure 4. Relationship of COIs to Measures Figure 5. Examples of MOEs and MOPs and Alignment to a COI Figure 6. System Evaluation Activities Figure 7. Site Selection Process

Version 1.1 1

1.0 INTRODUCTION

1.1 TSA T&E MISSION

The T&E process within the Transportation Security Administration (TSA) measures perfor-mance levels of new technologies and integration of the technologies as part of security opera-tions. The T&E results are used to inform the stakeholders as to whether the new system is ready for deployment by leveraging the best of breed T&E methodologies and techniques.

The TSA is responsible for maintaining and improving the performance and effectiveness of ap-proximately 760 checkpoints, 2,100 X-ray units, 1,500 explosives trace detectors, 2,000 en-hanced metal detectors and supporting physical access control at over 460 airports around the country. The TSA is also responsible for identifying new technologies for screening air cargo, cargo containers that transit through U.S. ports and other surface transportation modes. In many cases one technology is applicable to several transportation modes. However, the same technolo-gy is typically developed by commercial manufacturers as a unique product supporting security screening requirements for a transportation mode or operational configuration for a variety of customers around the world. TSA analyzes these commercial products to identify candidate products to improve security screening. Overall life cycle costs can be minimized by leveraging these commercial products. To this end, it is critical for the TSA to verify and validate that these commercial products provide capabilities required in TSA operational environments.

1.2 OBJECTIVES OF THE GUIDEBOOK

Described in this Guidebook are T&E best practices for developing an integrated T&E strategy within the full system engineering life cycle (SELC). Strategies and best practices from a variety of sources were leveraged to develop a methodology tailored to the specific needs of TSA.

TSA is a relatively new administration within the U.S. government that has and continues to have many aggressive milestones for implementing improvements to security screening opera-tions in a variety of transportation modes. Over the past seven (7) years, TSA has improved T&E capabilities within a variety of areas. More recently, a couple major changes within the Depart-ment of Homeland Security (DHS) and TSA provide an opportunity to further improve the TSA T&E strategy and methodology.

A new TSA Systems Integration Facility (TSIF) is now available that when used properly can better ensure readiness for Operational Testing (OT). The TSIF provides the capability to con-struct operationally realistic environments for use by T&E entities to discover and eliminate li-mitations and inefficiencies prior to implementation at airports and other operational environ-ments.

The DHS has drafted and implemented a new Acquisition Directive (AD) 102-01 that provides detailed policies and practices for the Acquisition Review Process (ARP) and SELC used to pro-cure, develop, deploy and monitor new capabilities that directly support the Department’s mis-sion. The AD 102-01 is an interim directive but is expected to be fully established in the near future. These new T&E policies describe oversight authorities, responsibilities, processes, and milestones. Additional management directives will be established to support and codify these policies for TSA.

This Guidebook provides a description of the best practices and processes to develop an inte-grated T&E strategy for all security technology improvement programs and a framework for im-plementing the strategy at TSA.

Version 1.1 2

1.3 SCOPE

The integrated T&E strategy described in this Guidebook focuses on best practices, techniques, processes and data used within the TSA OST for T&E of security systems for air passenger screening, carry-on baggage screening, checked baggage screening, and cargo screening within the aviation and surface transportation modes.

T&E strategies, techniques and methods are described that support the activities across the full SELC to include identification of need, analysis and solution selection, obtaining the solution through procurement, deployment and support as described by AD 102-01.

Most of the security systems TSA procures are products that have been developed commercially for a variety of customers around the world with similar needs as TSA. Typically, the TSA does not contract a security device manufacturer to design and develop a new system. Instead, the TSA investigates existing products, requests manufacturers to consider incorporating desired ca-pabilities and identifies candidate commercial products for potential integration into TSA’s exist-ing security operations. The Department of Defense (DoD) conducts this type of acquisition for some military equipment and has established T&E practices for what is called Non- Developmental Item (NDI) systems. This Guidebook leverages best practices from these NDI practices as well as other traditional developmental and operational T&E strategies from DoD, DHS, commercial and international organizations.

1.4 ROLE OF T&E FOR TSA ACQUISITION

T&E plays a significant role and importance within the TSA acquisition process. The impact of making a change to existing security screening operations around the country is felt by more than 43,000 trained and certified TSOs employed by TSA, the two (2) million passengers that fly on a daily basis, the approximately 460 airports around the country, the airlines that fly through these airports and air cargo operators, as well as participating rail terminals, mass-transit terminals, and port terminal operations. An effective and efficient T&E strategy is essential to identify limita-tions and resolve inefficiencies as early as possible. This ensures that any disruption to the opera-tional environment is minimized. The TSA program forms an Integrated Product Team (IPT), a T&E Working-level Integrated Product Team (WIPT), and System Evaluation Team (SET) to manage the developmental and operational evaluation of all systems. The TSA Evaluation and Operational Integration (E&OI) Section leads the SET and coordinates with the T&E WIPT to evaluate test results and data gathered for mission performance and operational suitability criteria to assess readiness of new systems and capabilities. The two major phases of T&E are Develop-mental Testing (DT) and OT as shown in Figure 1.

Version 1.1 3

Figure 1. T&E Phases within the Acquisition Life Cycle

DT is conducted early in the SELC to verify security device manufacturer’s assertion of perfor-mance against TSA Critical Technical Parameters (CTP), other technical requirements, and to demonstrate readiness for OT. CTPs are defined to support assessment of Critical Operational Issues (COI). Qualification Testing (QT) is conducted during DT to validate that a device meets minimum CTP thresholds and all technical requirements. Certification Testing (CT) is con-ducted, as necessary, to validate that a device meets all requirements to operate within a specific environment. The basic question answered by DT is “does the system work as designed?”

DT is completed to ensure, at a minimum, the equipment is safe and meets desired CTP thre-sholds so that OT activities can focus on equipment that show the most potential. OT verifies and validates the equipment against the Operational Requirements Document (ORD) and for opera-tional scenarios defined in the Concept of Operations (CONOPS). OT is a critical phase that as-sesses the system’s readiness for procurement and deployment in the production environment.

Measures of Effectiveness (MOE) and Measures of Suitability (MOS) are used as the primary evidence for evaluation of COIs. Other Follow-on Operational Test and Evaluation (FOT&E) activities are conducted to monitor the performance of the system, once in production, to ensure it continues to meet the desired performance levels. The basic question answered by OT is “does the system design work as intended?”

The sections that follow describe the strategies, techniques and best practices needed to develop an integrated T&E strategy for all TSA security technology programs.

Events

Plans

Evaluation / Reports

Evaluation Strategy

TEMP and SEP TEPs TEPs

SERTER /

ERB

Need Analyze / Select Obtain

Produce / Deploy / Support

ADE 1 ADE 2A ADE 3

ADE 2B

DEVELOPMENTAL QT/CT

LUTOPERATIONAL IOT&E iSAT

OTHER SOURCES

TER /

ERB

FRPOTRR3

MODELING & SIMULATION

DTRR

FOT&E

Vendors Ind Gov’t Labs

FAT/SAT

Version 1.1 4

2.0 ROLES AND RESPONSIBILITIES

This section outlines the roles and responsibilities of the various stakeholders and support ele-ments. Additional roles and responsibility detail is provided in Appendix P – T&E Responsibili-ty and Assignment Matrix.

2.1 INTEGRATED PRODUCT TEAM (IPT)

A Project Lead representing the Program Management Office (PMO) chairs the IPT. Early in the strategic planning process, the Project Lead establishes an IPT that meets regularly to plan and coordinate all acquisition activities.

2.2 T&E WORKING-LEVEL INTEGRATED PRODUCT TEAM (T&E WIPT)

The PMO chairs the T&E WIPT. Typical participants include E&OI, TSIF, Transportation Secu-rity Lab (TSL), Office of Security Operations (OSO) (including Office of Technical Training (OTT)), project Contracting Officer’s Technical Representative (COTR), and others as appropri-ate. As needed, representatives from support organizations may be asked to participate in the

T&E WIPT.

The T&E WIPT will:

• Provide a forum for all required organizations to be involved early in the integrated T&E strategy development process.

• Identify issues early and leverages available information and lessons learned to improve the acquisition and T&E processes.

• Develop an integrated T&E strategy, as well as a coordinated program approach for DT, OT, and Modeling and Simulation (M&S) to support a determination of a system’s operational ef-fectiveness and suitability.

• Develop a high-quality Test and Evaluation Master Plan (TEMP).

• Assist in the development of the integrated T&E strategy and COIs.

• Review and provide inputs to the acquisition strategy, Operational Requirements Document

(ORD), Request for Proposal (RFP), Statement of Work (SOW), Procurement Specification (PS), training plans, system threat assessment, System Evaluation Plan (SEP), and Test Event Plans (TEP).

2.3 SYSTEM EVALUATION TEAM (SET)

A SET is established to evaluate a new system or capability that is subject to the ARP, as re-quested by the PMO. The SET Chair is appointed by the OTA and is responsible for developing the System Evaluation Report. The SET consists of members from the E&OI, TSIF Sections and the TSL. The SET coordinates with representatives from the PMO, T&E WIPT, user communi-ty, and acquisition communities. Each of the primary (OST E&OI/OST Engineering/DHS S&T TSL) and associated (PMO/OSO/OTT/Logistics/etc.) organizations will nominate a single SET representative to ensure stakeholder interests are communicated and preserve equities in the overall testing and evaluation process. The primary objective of the SET is to evaluate the effec-tiveness and suitability of the system under test through integrated T&E. For the SET to be suc-cessful, members must be empowered by their parent organizations to make decisions, and keep their organization informed of issues discussed by the SET. The goal of the SET is to produce a

Version 1.1 5 single, comprehensive, and credible assessment of system performance throughout the SELC, thereby reducing T&E costs and reducing time to procure and field viable systems.

The SET will:

• Lead development of the integrated T&E strategy, consistent with program manage-ment/acquisition objectives and schedule.

• Support development of the TEMP, most notably sections C and D, and appendices (i.e., Re-quirements Crosswalk Matrix (RCM) and Verification Requirements Traceability Matrix (VRTM)), and other acquisition, requirements definition, and test and evaluation support do-cumentation.

• Identify and communicate T&E resource requirements, including units under test, support equipment, operator training, site enhancements, etc.

• Create developmental and operational testing (DT/OT) entrance and exit criteria, monitor progress toward achievement, and provide input to test readiness reviews.

• Ensure credible threat portrayal during DT/OT consistent with system requirements as de-fined in the ORD and Functional Requirements Document (FRD), including planning for threat surrogate development and Verification, Validation & Accreditation (VV&A) as re-quired.

• Establish criteria for selecting laboratory and field sites for DT/OT events, consistent with integrated T&E strategy requirements.

• Support the development of COIs with the user community/proponency.

• Develop the SEP.

• Prepare the Emerging Results Brief (ERB).

• Prepare and brief the System Evaluation Report (SER).

• Review draft system requirements for testability.

2.4 IPT/T&E WIPT/SET RELATIONSHIP

The PMO establishes an IPT to plan, manage and execute all acquisition activities for each new system. The E&OI section establishes a SET to plan, manage and execute all T&E activities within the project (exceptions may include Air Cargo Screening Program (ACSP) because projects are not a formal acquisition program). The SET develops the integrated T&E strategy and coordinates with the IPT to ensure all test entrance criteria are met. The IPT works closely with the SET to coordinate efforts in support of the acquisition program.

At the program level the T&E WIPT coordinates all project T&E issues. Issues that cannot be resolved within the SET are raised to the T&E WIPT. The T&E WIPT provides a central forum to share lessons learned and establish a consistently efficient and effective T&E program.

2.5 PROGRAM MANAGEMENT OFFICE (PMO)

The PMO is responsible for supervising all aspects of the system acquisition lifecycle. With re-spect to T&E, the PMO will:

• Establish the program acquisition strategy, determines the budget, ensures sufficient re-sources are available for testing and evaluation purposes, and aligns priorities and activities to meet program objectives.

• Develop and maintains the TEMP.

Version 1.1 6

• Assist in the coordination of DT/OT resources, documentation, facilities, and support re-quired for each project.

• Ensure the integration of T&E activities in the master schedule and project plan, monitor progress, and review project test plans and reports.

• Serve as the business communications liaison between TSA and the equipment vendor.

2.6 DHS S&T TRANSPORTATION SECURITY LABORATORY (TSL)

The TSL is the test laboratory and is responsible for the statutory Explosives Detection Systems (EDS) Certification Testing (CT) as well as qualification testing for other programs. The CT Program is reserved for detection and testing of bulk and trace explosives detection systems and equipment. The focus is on providing laboratory certification of explosives detection equipment, certifying that salient performance characteristics, such as the probability of detection of all cate-gories of explosives with appropriate false alarm rates and throughput rates are met. The TSL roles and responsibilities include:

• Review all requirements to ensure they are traceable to the ORD and FRD and are consistent, achievable, testable and measurable.

• Pre-certification testing.

• Certification Testing.

• Qualification Testing.

• Producing Qualification Data Package Audits.

• Participation in the SET.

• Coordination with the DT/OT communities in development of SEPs and TEPs.

• Assisting with development of system requirements documentation.

• Assisting the program management/DT/OT communities during development of the inte-grated testing and evaluation strategy as codified in the TEMP, and serve as concurring au-thority.

2.7 OST TSA SYSTEMS INTEGRATION FACILITY (TSIF)

The TSIF enables the TSA OST to evaluate transportation security systems, support infrastruc-ture, and mission profiles in a controlled environment. The TSIF provides the capability to ex-ecute tests in an operationally representative setting without impacting the operations of an ac-tual airport. The TSIF consists of a fully functional baggage handling system to support checked baggage screening systems, an open area with the capability to test cargo and vehicle screening systems, and an entire area devoted to passenger screening systems. The TSIF bridges the capa-bilities gap between the developmental/certification testing conducted at the DHS S&T TSL and the operational testing conducted by TSA E&OI. The TSIF roles and responsibilities include:

• Review all requirements to ensure they are traceable to the ORD and FRD and consistent, achievable, testable and measurable.

• Qualification Data Package Audits.

• Developmental Testing.

• Qualification Testing.

• Active participation in the SET.

• Coordinate development of TEPs.

Version 1.1 7

• Serve as an authority on aviation security processes and procedures.

• Assist with development of system requirements documentation.

• Assist with development of system CONOPS and associated enabling Standard Operating

Procedures (SOP) and Deployment Operating Plans (DOP), as required.

• Coordinate with Office of Inspection (OI), Aviation Screening Assessment Program (ASAP) and TSL on threat representation as part of developmental and operational testing.

• Assist the program management/testing and evaluation community during development of the integrated testing and evaluation strategy, as codified in the TEMP, and serve as concur-ring authority.

• Participate in Operational Test Readiness Reviews (OTRR) prior to commencing system op-eration and record testing.

• Develop and sponsor (in conjunction with the OST and other stakeholders) concept demon-strations and system enhancement/process optimization initiatives.

• Advise on potential doctrinal, organizational, training, manpower, personnel, logistics, and facilities impacts associated with security equipment deployment.

2.8 OST EVALUATION & OPERATIONAL INTEGRATION (E&OI)

The E&OI Section is charged with providing robust, credible, and independent system evalua-tions to the PMs, PMOs, DHS Director of Operational Test and Evaluation (DOT&E), and key acquisition decision makers. The E&OI section operates on three overarching principles: early involvement, continuous evaluation, and independent T&E. The responsibilities for E&OI in-clude:

• Conducting OT&E.

• Reviewing all requirements documentation to ensure they are traceable to the MNS, ORD, and FRD and consistent, achievable, testable and measurable.

• Active participation in the SET.

• Preparing the SEP and the SER.

• Providing TEMP inputs, most notably Part D (OT) and resource requirements in Part E.

• Reviewing DT test plans and reports.

• Ensuring all TEPs adequately support the T&E strategy.

• Assisting in the development of COIs and measures.

• Providing the PMO T&E resourcing and schedule requirements.

• Monitoring system developmental progress throughout the SELC.

• Serving as the chairs for the SETs.

• Providing a recommendation as to the readiness of the system prior to beginning IOT&E.

2.9 OFFICE OF SECURITY OPERATIONS (OSO)

OSO is responsible for providing a passenger and checked baggage screening function at airports and is considered the end user of screening systems. As the principle user representative to IPT/T&E WIPT/SET, the general T&E responsibilities for OSO are to:

• Serve as the primary subject-matter experts on aviation security process and procedures.

• Assist with the development of system requirements documentation.

Version 1.1 8

• Lead the development of COIs and associated rationale.

• Lead the development of system CONOPS and associated enabling SOPs, DOPs, training plan and documents.

• Coordinate with the DT/OT communities on test threat representation.

• Assist the program management and T&E community during development of the TEMP, and serve as concurring authority.

• Provide requested inputs to T&E documentation.

• Assist with the OT site selection and coordination.

• Assist with securing additional support personnel via the National Deployment Office (NDO) to support DT/OT.

• Participate in the OTRRs.

• Develop and sponsor, in conjunction with the OST and other stakeholders, concept demon-strations and system enhancement/process optimization initiatives.

• Provide advice on potential doctrinal, organizational, training, manpower, personnel, logis-tics, and facilities impacts associated with the security equipment deployment.

• Lead post-deployment process oriented assessments.

2.10 OST ENGINEERING

The overall mission of the OST Engineering Section is to provide for system design, analysis and technical support of OST's electronic screening systems (ESS) throughout their lifecycle. OST Engineering is primarily responsible for ensuring that proper requirements are defined for each critical TSA acquisition decision. These requirements are described in the following documents that serve as inputs to the integrated T&E planning process:

• CONOPS.

• ORD.

• FRD/Procurement Specification.

• Verification Requirements Traceability Matrix (VRTM).

Additional OST Engineering roles that may impact the integrated T&E Process include the fol-lowing:

• Coordinates developmental test activities with the TSL and TSIF.

• As a member of the IPT, participates in the decision making process, including providing recommendations for milestone decision points.

2.11 OVERSIGHT AUTHORITIES

The oversight authorities as described in the DHS MD 026-06 Test and Evaluation, establishes the roles and responsibilities for all T&E activities within DHS and its components. The appli-cable organizations in MD 026-06 that are included in this guidebook are the DHS Test and Evaluation and Standards Division (TSD), Program Manager, and the Operational Test Agent.

The roles and responsibilities of the Program Manager are described in the Program Manage-ment Office (PMO) section. The Operational Test Agent is described in the OST Evaluation & Operational Integration (E&OI) section because E&OI acts as the TSA Operational Test Agent.

In addition, the roles and responsibilities of TSA’s Acquisition and Program Management Sup-

Version 1.1 9 port Division (APMSD) are included to accomplish the roles and responsibilities of the TSA Ac-quisition Executive.

2.11.1 DHS S&T Test & Evaluation and Standards Division (TSD) TSD works across all DHS elements and entities to support T&E programs and Standards De-velopment efforts. The Office of Test and Evaluation establishes and manages DHS T&E policy and procedures and to efficiently, effectively and objectively coordinate T&E resources to verify attainment of technical performance specifications and operational effectiveness and suitability.

In this role, TSD performs the following functions:

• Develops and manages DHS T&E policy and procedures.

• Advises the Component Program Managers in developing T&E documentation, planning for the T&E and in the resolution of T&E issues.

• Reviews the Mission Needs Statement (MNS), CONOPS, Integrated Logistics Plan, ORD and associated COIs.

• Approves the Test and Evaluation Master Plan (TEMP).

• Approves the Operational Test Agent (E&OI serves as the Operational Test Agent for TSA).

• Participates in the IPT.

• Reviews test plans, test reports, and observes testing to maintain knowledge of Developmen-tal Testing and/or Contractor Testing.

• Approves Operational Test Plans.

• Participates in Operational Test Readiness Reviews (OTRR).

• Facilitates use of non-DHS facilities for testing and promotes accreditation of T&E capabili-ties as appropriate.

• Reviews OT Reports.

• Participates in Program and Design Reviews.

• Chairs the DHS T&E Council.

• Authors the Letter of Assessment of the OT Report as appropriate.

2.11.2 Acquisition and Program Management Support Division (APMSD) The TSA Office of Acquisition’s APMSD is a leading source of acquisition knowledge and in-formation. APMSD serves or designates the TSA Acquisition Executive for TSA on all pro-grams and as a resource to support TSA program management activities throughout the acquisi-tion process. Key activities for APMSD are to:

• Provide acquisition assistance and advice to TSA program management.

• Guide TSA leadership through the acquisition process to successful navigate all acquisition decision events.

• Maintain the corporate knowledge base of acquisition information and programs.

2.12 OTHER ORGANIZATIONS

Other organizations that may be included as part of the T&E team are:

• Office of Public Affairs (OPA).

• Office of Chief Counsel (OCC).

Version 1.1 10

• Office of Safety, Health, and Environment (OSH&E).

• OST Integrated Logistics Support (ILS).

• OST Independent Verification and Validation (IV&V).

• Office of Inspection (OI).

• OST Security Technology Integration Program (STIP).

• Others as appropriate.

Version 1.1 11

3.0 INTEGRATED TEST AND EVALUATION

3.1 OVERVIEW

To improve security screening operations the TSA Office of Security Technology (OST) identi-fies new security devices and capabilities, develops appropriate concepts of operations, assesses alternatives, defines requirements, identifies candidate manufacturers, conducts qualification testing, evaluates the impact of changing the operational environment, procures, deploys and maintains the equipment in the field. This is a process conducted for each new and significant modification to existing technology and involves participants from a wide range of organizations from within TSA programs, security operations, laboratories, T&E entities, and commer-cial/private organizations impacted by changes to security screening operations.

Efficiently and effectively coordinating system evaluation activities throughout the SELC re-quires all entities involved to have a good understanding of the process and a common approach.

Entities involved in system evaluation include laboratories (DHS TSL, Johns Hopkins University Applied Physics Laboratory (APL), Sandia National Laboratories, Tyndall Air Force Base, and other relevant laboratories), TSA OST program offices, TSA OSO, TSA OST Surface, TSA E&OI, TSA OST Engineering, DHS ARP authorities and a variety of support contractors. Each entity has T&E responsibilities for each new project. Conducting an efficient and effective sys-tem evaluation requires developing an integrated T&E strategy early in the SELC.

The major system evaluation objective is to determine whether the system is:

• Operationally Effective.

• Operationally Suitable.

The system contribution to mission capability is assessed based on data that supports these two evaluation domains.

The integrated T&E strategy is the product of the T&E WIPT and SET working together to de-velop one approach that balances the needs and objectives of the Program, user community, Ac-quisitions, and T&E. The T&E WIPT prepares a TEMP in accordance with the integrated T&E strategy that is a “top-level” planning document for all T&E related activities of major acquisi-tion programs (i.e. “what” are we going to test). The TEMP describes the necessary DT and OT activities that need to be conducted to evaluate system technical performance, and operational effectiveness/suitability. The COIs must be identified in the TEMP. Concurrently during TEMP development, the SET prepares a SEP that focuses on the T&E activities and method of assess-ment necessary to fully evaluate the operational effectiveness and suitability requirements (i.e.

“how” are we going to test it).

The SER identifies capabilities and limitations of the system under test and recommended im-provements. Capabilities and limitations are addressed with respect to the current system and concept of operations that identify impacts to security operations. Guiding principles for inte-grated test and evaluation are to:

• Provide essential information to support Acquisition Decision Events (ADE) and other key programmatic decision events.

• Provide continuous feedback for assessing security system technical and acquisition risk.

• Verify the attainment of technical performance specifications for security systems and sub-systems under test.

Version 1.1 12

• Discover issues at the earliest opportunity so they may be addressed and resolved with the least impact to program schedule and cost.

• Support the formulation of realistic system requirements and specifications and ensure the system performance risks are acceptably minimized.

• Reduce test time and cost through comparative analyses, data sharing, and use of all data sources for evaluation.

• Verify the corrective action process during system development.

• Provide assessments of system capabilities after deployment, when requested.

• Provide operational utility assessments of new technologies for transition to the formal ac-quisition process.

COMBINED AND CONCURRENT EVENTS

The integrated T&E strategy may include combining DT and OT to enhance efficiency, cost less or improve the effectiveness. Combining or integrating DT and OT events maybe worthwhile if the data collected during DT fits the needs and criteria of OT.

The integrated T&E strategy may include a variety of DT events, OT events, modeling, simula-tion, and analysis (MS&A) and other system specific evaluation activities. The integrated T&E strategy supports the system evaluation with the minimum essential testing to generate the re-quired data that facilitates learning, assesses technology maturity and interoperability, facilitates integration into fielded operations, and confirms performance against documented needs and re-quirements. The objective of the integrated T&E strategy is to develop a synergy among all planned events to provide information that the system evaluator uses to prepare reports with re-quired information for the decision maker.

COTS SECURITY DEVICES

The majority of systems that TSA selects for evaluation are Commercial-Off-The-Shelf (COTS) security devices. TSA must ensure each COTS device satisfies its need. To this end, TSA con-ducts domestic and international market research, publishes Requests for Information (RFI) to identify new technologies and capabilities, invites manufacturers to participate in early demon-strations during concept development, and collaborates with national and international organiza-tions that are interested in improving current requirements and advancing future needs. The TSIF provides additional opportunities for testing new equipment and upgrades within a simulated op-erational environment as part of DT and OT without disrupting the airport environment. The TSIF provides a controlled environment for assessing technical conformance and operational ef-fectiveness/suitability during concept development, DT and OT. This will minimize the disrup-tion in the operational environment and provide more opportunities to conduct operationally rea-listic tests during all phases of T&E and better ensure that systems meet their intended use prior to procurement and deployment. These activities need to be incorporated into the integrated T&E strategy.

3.2 EVALUATION STRATEGY DEVELOPMENT

Developing the integrated T&E strategy occurs early in the acquisition process and in coordina-tion with appropriate elements of the Program, user community, Acquisitions, and T&E ele-ments. The MNS expresses the need for the system capabilities while the ORD expresses the ca-pability requirements that satisfy the need. The major focus of system evaluations is the COIs

Version 1.1 13 that are evaluated against the ORD. The T&E WIPT and SET coordinate their activities to de-velop the TEMP and SEP, respectively, as illustrated in Figure 2.

Figure 2. TEMP and SEP Development Process

Much of the work in developing the integrated T&E strategy depends on requirements that are accurate, testable, and reflect a complete system that can operate in the intended environment by normally trained personnel. All operational requirements and KPPs are documented in the ORD while functional/technical requirements are documented in the Functional Requirements Docu-ment (FRD), which is part of the PS. Other requirement documents may be referenced that spe-cify functionality for interoperating with other systems. Though the CONOPS does not contain specific requirements, it is vital for providing a complete understanding of the intended use of the system. Other activities and products such as Acquisition Strategy, Analysis of Alterna-tives/Alternatives Analysis (AoA/AA), integrated logistics support plan, and system threat as-sessments will contribute to the development of the integrated T&E strategy. The T&E commu-nity participates in these requirements development activities to ensure all requirements are test-able and that the integrated T&E strategy reflects all key system operational issues (i.e., COIs).

The ORD, PS/FRD, CONOPS, and COIs are required for the development of the TEMP and SEP. As the requirements evolve and mature, so will the integrated T&E strategy that drives the “what” in the TEMP and the “how” in the SEP.

DIFFERENCES BETWEEN DT AND OT

Each T&E event provides data that supports validation of technical and/or operational require-ments. While DT is oriented toward conformance with technical requirements and is typically conducted under controlled conditions in laboratory, simulated environments or vendor facilities, OT verifies operational requirements (i.e. the ORD) and mission success with typically trained operators within operational environments such as airports, mass transit facilities, or other opera-tional sites. Typically, DT verification is based on requirements documented in the Procurement Specification. OT validation derives its test requirements from COIs and KPP requirements in the ORD while DT verifies CTPs and requirements in the FRD.

3.2.1 Developing COIs

The goal of T&E activities is to ensure that new or modified systems meet users’ operational needs. The two key elements of success in this process are to:

• Ensure that user requirements for the new system’s operational capabilities are clearly and accurately expressed.

• Ensure that these operational requirements are understood, confirmed to be testable, and sub-sequently verified by the tester through a structured test program.

SET

Inputs

MNS, ORD,

CONOPS, PS,

COI, KPP,

Acquisition Strategy, CTP

Develop Initial

Integrated T&E Strategy

Finalize Integrated

T&E Strategy

Develop Data Source

Requirements

Final Coordination of the Integrated

T&E Strategy

T&E

WIPT TEMP

SEP

project

PROGRAM

Version 1.1 14

COIs are phrased as questions that must be satisfactorily resolved to determine system effective-ness and suitability. They address essential capabilities, risks, or uncertainties that must be eva-luated. COIs are developed cooperatively by the functional proponent (user community repre-sentative), the PM, and System Evaluator utilizing the ORD, FRD and CONOPS. Though each security technology acquisition project is different, common evaluation areas clearly emerge across all projects within operational effectiveness and suitability domains that can be used to identify COIs and define MOE/MOSs and MOPs. Figure 3 provides a list of common evalua-tion areas that may be used to identify and define COIs and corresponding MOE/MOSs for sys-tem evaluation.

The COI should specify the appropriate and specific environment such as an airport checkpoint environment. The common evaluation areas may serve as good COIs but other issues should also be considered that may be specific to the technology, system or concept of operations. Well con-structed COIs manifest traits that are:

• Key operational concerns of the decision maker, with bottom line standards of performance that signify that the system is operationally ready to proceed at ADE 3.

• Operationally relevant, mission focused issues.

• Overarching, total operational system measures.

• Stated to clearly reflect why the system is being acquired.

Figure 3. Common Evaluation Areas

Version 1.1 15

• Not pass/fail absolutes but are “show stoppers”. A system falling short of the criteria should not proceed at ADE 3 unless convincing evidence of its operational effectiveness and suita-bility is provided to the decision makers; and

• Few in number, reflect total operational system concerns, consider system maturity, and em-ploy higher order measures.

Each COI is associated with one or more MOE (if it is under the operational effectiveness do-main) or MOS (if it is under the operational suitability domain). These MOE/MOSs serve as the measures used to evaluate the systems ability to satisfy the COI. An MOE/MOS is a quantitative or qualitative measure of performance or suitability. KPPs are assigned as an MOE/MOS that directly support the evaluation of a COI. Characteristics of good MOE/MOSs are:

• Inclusive of all system Key Performance Parameters (KPP).

• Achievable and can be evaluated.

• Provide clear guidance on conditions applicable to measuring each criterion and for scoring the results.

• Avoid terms that could be misinterpreted doing the analysis and/or system evaluation- “bet-ter, greater than, less than, at least as good as,” etc.

• Reflect the minimal system acceptable performance for entry into procurement.

• Accountable for TSO/traveling public safety.

In many cases an MOE may be derived from various elements or factors that drive the perfor-mance or suitability of the operational environment called Measures of Performance (MOP).

Figure 4 shows the hierarchical relationship such that MOPs are used to derive MOE/MOSs that form the criteria for evaluating and assessing COIs.

Figure 4. Relationship of COIs to Measures

CRITICAL OPERATIONAL ISSUE (COI)

MEASURE OF EFFECTIVENESS (MOE) or

MEASURE OF SUITABILITY (MOS)

MEASURE OF PERFORMANCE (MOP)

• May be evaluated based on one or more MOE/MOSs

• The MOE/MOS criteria is typically specified as a threshold

• A threshold may be stated as part of the MOE/MOS statement and derived based on one or more MOPs

• KPPs are typically addressed by MOE/MOSs

• MOPs are derived or evaluated based on a collection of required data elements

MEASURE OF EFFECTIVENESS (MOE) or

MEASURE OF SUITABILITY (MOS)

MEASURE OF PERFORMANCE (MOP)

Version 1.1 16

MOE/MOSs and MOPs may require collecting data from a variety of data sources from DT, OT, M&S, data analysis, vendor testing, industry and international test results and analysis or any other validated source that may provide information that supports the effective evaluation and assessment of system performance and suitability. An MOP is almost always a measure of a sys-tem-specific design or performance characteristic. MOPs should not be stated as a single-word (e.g., accuracy). Figure 5 provides an example of MOEs and MOPs that may be used for the common evaluation area (1) mission performance effectiveness.

Figure 5. Examples of MOEs and MOPs and Alignment to a COI It is critical to have an understanding of COIs, KPPs, mission and program objectives to achieve an effective evaluation. Toward this end, developing a Baseline Correlation Matrix (BCM) will promote an understanding of both the requirements of the system, critical issues, KPPs and their relationships. It also supports requirements validation by mapping requirements, KPPs, CTPs, MOEs/MOSs, MOPs and COIs to ensure all requirements are addressed during DT and/or OT.

The BCM may be used to derive a variety of Requirements Traceability Matrices (RTMs) such as the VRTM and RCM.

Pd for explosive A on the body

Pd for explosive A in a bag

Pd for IED A on the body

Pd for IED B in a bag

MOPs

Minimum probability of detection of the checkpoint against explosives is > 80%

Does the system perform effectively in the airport checkpoint environment?

> 80% criteria Minimum probability of detection of the checkpoint against weapons is > 75% > 75% criteria

MOEs

COI

Pd for weapon A on the body

Pd for weapon A in a bag

Pd for weapon B on the body

Pd for weapon B in a bag

Uptime

COI

MOSs criteria

Will the system be reliable, maintainable, and available in the airport checkpoint environment?

Uptime criteria criteria

Mean Time Between Failures (MTBF) should be at least 60 hours >= 60 hours

Operational Availability (AO) should be 97% or higher >= 97%

Mean Time To Repair (MTTR) should not exceed 1 hour <= 1 hour

MOPs # of Failures

Down-time

Time to Diagnosis

Time to Fix

Return to Service

Version 1.1 17

3.2.2 Evaluation Planning

Evaluation planning is conducted “top-down” while evaluation execution is conducted “bottom-up.” The system evaluation process makes use of data from all credible sources, including ana-lyses, contractor testing, developmental testing, operational testing, MS&A, and demonstrations to create an integrated and continuous flow of information on the status of a system’s capability.

It applies to all types of acquisition strategies and all categories of acquisition programs. It is evolutionary throughout all acquisition phases, including the Produce / Deploy / Support acquisi-tion phase.

Planning for system evaluation is accomplished early in the Analyze / Select acquisition phase.

The TEMP is the T&E contract between all stakeholders that establishes the executive level strategy and is the primary T&E planning document for system acquisition. The TEMP includes the scope, criteria and rationale for the COIs and includes the RCM and VRTM. A SEP is created to implement the evaluation activities necessary to fulfill the TEMP. The SEP includes the Data Source Matrix (DSM) and the BCM. The data sources used to support evaluation of each COI are identified in the DSM.

Data sources may be any test event, research, MS&A or any other event that collects data elements needed for evaluation of COIs. Each T&E event contributes data that is required for the evaluation. These activities are coor-dinated with the IPT, T&E WIPT and SET during the process of developing the integrated T&E strategy.

The integrated T&E strategy will em-ploy only those events necessary to collect data needed to evaluate COIs and verify attainment of all require-ments. All DT and OT activities are captured in the SEP. TEPs and Test Event Reports (TER) are generated for each test event. Figure 6 illustrates this structure.

DEVELOPMENTAL TEST

TSA conducts DT to evaluate new concepts, identify candidate manufacturers, qualify systems, and gauge system readiness for OT. The primary objective of DT is to validate that all technical requirements have been met. Other objectives are to verify that the system is fully integrated, stable, and has no adverse effect on the systems or system components operating in or near the area. It may include one or more demonstrations to ensure that the system meets user and opera-tional requirements. DT provides information about risk and risk mitigation, assesses the at-tainment of technical performance parameters, provides empirical data to validate models and simulations and provides information to support periodic technical performance and system ma-turity evaluations. The DT process will differ depending on the size and complexity of the sys-tem. It should progress from verifying requirements at the unit level and incrementally to the

Figure 6. System Evaluation Activities

TSL E&OITSIF

TEPs

• Independent Labs

• Vendors

• M&S

Verify attainment of all requirements

TERs

TEMP

VRTM

RCM

SER

SEP BCM

DSM

Version 1.1 18 fully integrated and fielded system, as the CONOPS and operational configuration evolves and matures. DT should involve the user and be conducted in laboratories, simulated environments or realistic operational environments. DT events may include integration testing; interoperability testing; technical conformance testing; and reliability, maintainability, and availability testing.

Typically, the PMO coordinates the DT activities in conjunction with OST Engineering and the TSL. After successful completion of DT, the program progresses to the OT phase. The OT team performs continuous evaluation during DT to leverage appropriate data sources and minimize redundant testing.

OPERATIONAL TESTING

OT is conducted to determine the operational effectiveness and suitability of a system under rea-listic conditions, against realistic threats utilizing typical, trained users. Whereas DT focused on the system itself, OT focuses on validating the system within the operational environment, against requirements in the ORD and FRD. OT events include Limited User Tests (LUT), Initial Operational Test & Evaluation, Integrated Site Acceptance Testing (iSAT), FOT&E and Cus-tomer Tests.