SharePoint RFP Tech - SLM 20Test 20Evaluation.pdf

Version 2.0 August 2005

Architecture Test and Evaluati on

The Test and Evaluation Plan of Immigration and Customs Enforcement

REVISION HISTORY

REVISION HISTORY

Version Date Summary of Changes

2.0 Added Section 2.0: Development Testing Plan

Updated Architecture T&E stakeholders and processes in alignment with the System Lifecycle Management (SLM) process and ICE Office of the Chief Information Officer (OCIO) organization Expanded Section 3.3: User Acceptance Testing (UAT) Facilitation Updated tools and test approaches to be consistent with Architecture T&E processes

Architecture Test and Evaluation Plan August 2005 i CONTENTS

CONTENTS

1.0 OVERVIEW

1.1 SLM Test and Evaluation Goals

1.2 SLM Test and Evaluation Stakeholders

1.3 SLM Testing Types

1.3.1 Development Testing

1.3.2 Functional Test and Evaluation

1.3.3 Technical Test and Evaluation

1.4 SLM Testing Documentation

1.5 Architecture T&E Services

1.6 Test Baselines

1.7 Architecture T&E Plan Maintenance

2.0 DEVELOPMENT TESTING PLAN

2.1 Unit Testing

2.1.1 Unit Testing Objectives

2.1.2 Unit Testing Process Description

2.2 Integration Testing

2.2.1 Integration Testing Objectives

2.2.2 Integration Testing Process Description

2.3 Functional Qualification Testing

2.3.1 Functional Qualification Testing Strategy

2.3.2 Functional Qualification Testing Process

2.4 Development Security Test and Evaluation

2.4.1 Development ST&E Objectives

2.4.2 Development ST&E Process Description

3.0 FUNCTIONAL TEST AND EVALUATION PLAN

3.1 System Acceptance Testing

3.1.1 System Acceptance Testing Strategy

3.1.2 System Acceptance Testing Process

3.2 System Security Testing

3.2.1 System Security Testing Strategy

3.2.2 System Security Testing Process

3.3 User Acceptance Testing Facilitation

3.3.1 User Acceptance Testing Strategy

3.3.2 User Acceptance Testing Process

4.0 TECHNICAL TEST AND EVALUATION PLAN

4.1 Integrated Performance Testing

4.1.1 Integrated Performance Testing Strategy

4.1.2 Integrated Performance Testing Process

i i CONTENTS

4.2 Interoperability Testing

4.2.1 Interoperability Testing Strategy

4.2.2 Interoperability Testing Process

APPENDIX A—ARCHITECTURE T&E TOOLS

APPENDIX B—REFERENCES

APPENDIX C—ACRONYMS AND ABBREVIATIONS

EXHIBITS

Exhibit 1: Key SLM Test and Evaluation Stakeholder Roles Exhibit 2: Testing Types Exhibit 3: Development Testing Strategies Exhibit 4: Functional T&E Testing Strategies Exhibit 5: Technical T&E Testing Strategies Exhibit 6: Development Testing Document Artifacts Exhibit 7: Architecture T&E Document Artifacts Exhibit 8: Key Architecture T&E Services Exhibit 9: Key Unit Test Objectives Exhibit 10: Key Integration Test Objectives Exhibit 11: Functional Qualification Testing Approaches Exhibit 12: Functional Qualification Testing Process Overview Exhibit 13: Functional Qualification Test Preparation Activities Exhibit 14: Functional Qualification Test Execution Activities Exhibit 15: Functional Qualification Test Analysis Activities Exhibit 16: Key Development ST&E Objectives Exhibit 17: System Acceptance Testing Approaches Exhibit 18: System Acceptance Testing Process Overview Exhibit 19: System Acceptance Test Preparation Activities Exhibit 20: System Acceptance Test Execution Activities Exhibit 21: System Acceptance Test Analysis Activities Exhibit 22: System Security Testing Approaches Exhibit 23: Test and Deployment System Security Testing Process

Overview Exhibit 24: Operations and Maintenance System Security Testing Process

Overview Exhibit 25: System Security Test Preparation Activities Exhibit 26: System Security Test Execution Activities Exhibit 27: System Security Test Analysis Activities

August 2005 Architecture Test and Evaluation Plan i i i CONTENTS

Exhibit 28: User Acceptance Testing Approaches Exhibit 29: User Acceptance Testing Process Overview Exhibit 30: User Acceptance Test Preparation Activities Exhibit 31: User Acceptance Test Execution Activities Exhibit 32: User Acceptance Test Analysis Activities Exhibit 33: Integrated Performance Testing Approaches Exhibit 34: Integrated Performance Testing Process Overview Exhibit 35: Integrated Performance Test Preparation Exhibit 36: Integrated Performance Test Execution Exhibit 37: Integrated Performance Test Analysis Exhibit 38: Interoperability Testing Approaches Exhibit 39: Interoperability Testing Process Overview Exhibit 40: Interoperability Test Preparation Exhibit 41: Interoperability Test Execution Exhibit 42: Interoperability Test Analysis

1 OVERVIEW

1.0 OVERVIEW

The Architecture Test and Evaluation Plan documents System Lifecycle Management (SLM) test and evaluation (T&E) strategies and processes, comprising Development Testing and Architecture T&E.

This plan outlines the key testing strategies and objectives of Development Testing and describes the Functional Qualification Testing (FQT) process.

For Architecture T&E, it details how the Immigration and Customs Enforcement (ICE) Office of the Chief Information Officer (OCIO) Technical Architecture and Integration (TA&I) Division independently validates system and infrastructure behavior and consistency with the technical architecture. Architecture T&E activities, processes, procedures, and strategies are identified; testing tools and documentation are described; and stakeholder roles are defined.

1.1 SLM Test and Evaluation Goals

In the SLM process, the goal of T&E is to reduce risk and increase the confidence of project stakeholders in the successful deployment of system and infrastructure projects. To achieve this goal, the SLM process includes:

Development Testing—Validates that components function properly, independent from one another and combined, and confirms that the system or infrastructure release is aligned with the design and requirements

Architecture T&E—Independently assesses and reports on the conformance to documented requirements and security measures, identification of deployment risks, compatibility of components with the production environment, and accuracy of system and component installation instructions

Development Testing and Architecture T&E provide ICE with a robust yet efficient means of determining the acceptability of system and infrastructure releases prior to implementation.

Note: Testing is performed to reduce risk, but cost and schedule constraints prevent the detection of every defect or deficiency. As a result, systems and infrastructures may be implemented with residual risk, which is the risk that remains when all known risks have been addressed.

1.2 SLM Test and Evaluation Stakeholders

The collaboration of many stakeholders is necessary to develop and execute effective testing strategies and provide a high level of test coverage for system and infrastructure projects. Exhibit 1 delineates key T&E stakeholders as well as their roles during T&E.

2 OVERVIEW

Exhibit 1: Key SLM Test and Evaluation Stakeholder Roles

Stakeholder Role

ICE Office of the Chief Information Officer

Provides high-level project management coordination and reporting, strategic planning, capital planning investment and control, and oversight for information security activities

Technical Architecture and Integration

Manages the Technical Architecture and Integration (TA&I) Program, which oversees the development of information systems through the System Lifecycle Management (SLM) process and advises the ICE Chief Information Officer (CIO) on technical recommendations and their associated impacts on the DHS enterprise architecture

Includes an Architecture Test and Evaluation (T&E) component that provides independent validation that systems and infrastructures meet functional, operational, security, reliability, performance, and interoperability specifications

Oversees the implementation of and adherence to the ICE SLM process;

manages the adoption, development, and specification of standards supporting the enterprise architecture; and monitors project adherence to ICE technical architecture standards

Includes a Functional T&E component that validates whether the final technology product meets the approved requirements and design specifications

Reviews the application of technologies to meet ICE business requirements and performance goals and presents guidance on the appropriate application of technologies to meet ICE strategic goals and objectives

Includes a Technical T&E component that validates and measures the viability, stability, and compatibility of system components and/or applications prior to live deployment

Data Management Provides stewardship of ICE data resources; implements their integration by providing the services necessary to provide smooth daily operations; ensures that resources are optimized; manages growth; and enforces data extraction, transformation, load, and delivery standards and functions

Serves as database specialists on Integrated Performance Testing Team, participates in testing effort (as needed), and analyzes test results

IT Service Delivery Directs operation of the enterprise IT infrastructure and provides IT support for ICE

Serves as network specialists on Integrated Performance Testing Team, participates in testing effort (as needed), and analyzes test results

Office of the Information Systems

Security Manager

Serves as the principal advisor to the ICE CIO for computer security matters and develops and oversees the ICE IT security program and reviews and approves the tools, techniques, and methodologies planned for System Security Testing

IT Project Manager

Serves as the central point of responsibility for project decisions and activities, coordinates the technical aspects of the project, and manages the project to achieve cost, schedule, and performance goals

System Owner

Serves as the project sponsor, representing the operational needs of the business unit and the system users and participates throughout the SLM process to ensure that the system meets operational and user requirements

Contractor Project Team

Develops and maintains information systems by following the ICE SLM process and by adhering to ICE technical architecture standards

Performs Development Testing, certifies the test lab configuration, and supports independent T&E

Designated Accrediting Authority

Assumes responsibility for operating a system at an acceptable level of risk and is accountable for the risk he or she accepts

Information System Security Officer

Ensures that appropriate steps are taken to implement information security requirements for information systems throughout the SLM process

Architecture Engineering

Architecture Assurance

3 OVERVIEW

1.3 SLM Testing Types

In the SLM process, Development Testing provides one type of testing, and Architecture T&E provides two types of testing: Functional T&E and Technical T&E. Exhibit 2 defines the testing types according to their testing strategies and responsible organizations.

Exhibit 2: Testing Types

Testing Type Testing Strategy Testing Organization

Development Testing Section 2.0

Unit Testing Integration Testing Functional Qualification Testing (FQT) Development Security T&E

Contractor Project Team

Functional Test and Evaluation Section 3.0

System Acceptance Testing System Security Testing User Acceptance Testing (UAT) Facilitation

Architecture T&E:

Functional T&E

Technical Test and Evaluation Section 4.0

Integrated Performance Testing Interoperability Testing

Architecture T&E:

Technical T&E

1.3.1 Development Testing

Development Testing typically comprises four testing strategies: Unit Testing, Integration Testing, FQT—which occurs after Unit Testing and Integration Testing, and Development Security Test and Evaluation (ST&E). Exhibit 3 defines these testing strategies.

Exhibit 3: Development Testing Strategies

Testing Strategy Description

Unit Testing Tests individual software units as part of coding during system development to ensure that unit functions conform to unit design and to ensure that every possible decision outcome in the code is made at least once. Unit Testing is complete when the programmer has produced code that successfully compiles, executes, and satisfies the criteria of the unit procedure.

Integration Testing Tests stand-alone software modules created from the integration of previously tested software units to validate proper unit integration, satisfaction of module design requirements for unit communication and interaction, and handling of and recovery from errors. Integration Testing retests individual software units to validate that they function properly after their integration.

Functional Qualification Testing

Tests the complete system to validate that the system satisfies documented requirements and that the system components operate and interact properly (Functional Qualification Testing [FQT] occurs after Unit Testing and Integration Testing.)

Development Security Test and Evaluation

Tests to ensure that the system security controls are in place, operating as documented, and satisfy security requirements

4 OVERVIEW

Development Testing may also include performance and/or interoperability testing and may be conducted by different testing organizations depending on the required type and scope of testing specified in the approved project work pattern.

1.3.2 Functional Test and Evaluation

Functional T&E determines whether the system or infrastructure satisfies the requirements described in the System Requirements Document (SRD).

Exhibit 4 defines the testing strategies for Functional T&E.

Exhibit 4: Functional T&E Testing Strategies

Testing Strategy Description

System Acceptance Testing

Verifies the functional requirements as described in the System Requirements Document to determine if the product performs the business functions as documented

System Security Testing

Validates implementation of security requirements and controls in the system and identifies potential intrusion or sensitive data exposure vulnerabilities

User Acceptance Testing Facilitation

Provides a controlled environment for actual users to test systems before the system becomes operational

For System Acceptance Testing (SAT) and System Security Testing, automated scripts, developed by Automation Engineers, can be used to identify any functional problems with the new release of the system. The automated scripts are run as a regression test to ensure that new code changes did not adversely affect the existing code.

1.3.3 Technical Test and Evaluation

Technical T&E evaluates the stability, capacity, response time, and throughput of a system by providing end-to-end Integrated Performance Testing and Interoperability Testing for ICE systems. Exhibit 5 defines the testing strategies for Technical T&E.

Exhibit 5: Technical T&E Testing Strategies

Testing Strategy Description

Integrated Performance Testing

Detects any performance and capacity limitations by generating system load that emulates the behavior of users conducting business transactions, determines when performance begins to degrade, and identifies bottlenecks across the system application and infrastructure

Interoperability Testing Assesses the compatibility and potential impact of multiple software components coexisting on the infrastructure

5 OVERVIEW

1.4 SLM Testing Documentation

During the SLM process, the project team records the results of lifecycle activities in SLM document artifacts. Exhibit 6 identifies the SLM document artifacts submitted by the project team as a result of Development Testing.

Exhibit 6: Development Testing Document Artifacts

Document Artifact Description

Development Test Plan Presents the activities to be performed in Development Testing Documents the testing scope and environment Describes the tests performed, including test controls, inputs, and outputs (includes test procedures as an attachment) Provides traceability to the requirements validated by Development Testing

Development Test Analysis Report

Documents the results of Development Testing Describes the system units and functions tested Evaluates the performance of the tested units and functions Analyzes the system capabilities demonstrated during Development Testing Identifies system deficiencies and any indicated improvements in system design or operation based on the results of Development Testing Presents determination concerning the readiness of the system to be turned over for independent test and evaluation (T&E)

Security Test and Evaluation Plan

Documents plan for assessing technical, operational, and management controls employed to mitigate risks Provides an overview of the functions of the system and its overall security posture as verified and described in the security test procedures and results Identifies security tests that will be conducted in order to validate security controls in place on the system and specifies verification methods

Development Security Test and Evaluation

Report

Provides an overview of the functions of the system and its overall security posture Explains how the security control validation testing was completed Documents the test results Documents analysis of security findings Provides recommendations to mitigate identified security findings

In preparation for independent T&E, Architecture T&E completes the following activities:

Evaluates the documents and code artifacts developed by the project team

Creates and delivers a Testing Options document and independent test plans for review by the System Owner and IT Project Manager

At the conclusion of independent T&E, Architecture T&E prepares a summary or report of the testing results, which is delivered to the project

6 OVERVIEW

team and other stakeholders. System Security T&E results are included in the Certification and Accreditation (C&A) Package developed by the Office of the Information Systems Security Manager (OISSM). Exhibit 7 identifies the SLM document artifacts provided by Architecture T&E for independent T&E.

Exhibit 7: Architecture T&E Document Artifacts

Document Artifact Description

Testing Options Offers multiple options for System Acceptance Testing (SAT) Specifies for each option whether automated and/or regression testing will be performed Describes which platforms (images and browsers) the application will be tested against for each option Identifies the number of business days estimated for testing to be completed for each option Summarizes risks that would be assumed by the System Owner and IT Project Manager for each option Identifies categorization of risk for each option Provides detailed description of testing that will be performed and that will not be performed for each option

Independent Test Plans

Documents the testing scope, methodology, and schedule Specifies the testing environment and configuration Identifies the roles and responsibilities Describes each scenario to be tested Defines the test procedures to be followed in conducting independent test and evaluation (T&E), which may include installation and configuration, code review, data management, system assurance, interoperability, performance, and load testing

Independent Test Summaries/Reports

Documents the testing scope and environment Summarizes the test results Provides traceability to the requirements validated by independent T&E

1.5 Architecture T&E Services

TA&I manages and performs Architecture T&E in a controlled Independent Testing Facility (located at 1120 Vermont Ave. NW, Washington, DC, 20005) that closely simulates the production environment.

Additionally, TA&I provides support and guidance to project teams before, during, and after independent testing. Exhibit 8 identifies the key teams within TA&I and the Architecture T&E services they provide for projects.

7 OVERVIEW

Exhibit 8: Key Architecture T&E Services

TA&I Teams Services

Functional T&E Provides independent assessment of developed system functional capabilities Offers testing options with varying scope and risk Develops System Acceptance Test strategy Prepares and conducts System Security Testing Documents and reports the testing results

Technical T&E Provides independent assessment of developed system operational characteristics Develops Integrated Performance Test strategy and executes performance tests Offers Application Profiling to quantify system communication requirements and properties Configures and installs applications into interoperability testing environment Prepares and conducts Interoperability Testing Documents and reports the results

Test Lab Engineers Sets up Web and application servers for independent T&E Configures and installs applications into performance and interoperability testing environments Applies application tuning modifications

Automation Engineers Develops and implements test automation procedures in support of SAT and System Security Testing Develops and maintains suite of automated regression scripts

Architecture Consulting and Engineering

(Technical Architects)

Provides application tuning services to identify and resolve issues affecting application performance and help optimize system performance

Application Integration Services

Sets up production-like Web and application servers for independent T&E

Quality Assurance Provides IT project managers with process assistance Performs System Lifecycle Management (SLM) document artifact assessments and brokers subject matter expert (SME) assessments

Configuration Management

Coordinates setup and management of document, code, change request, and test problem report configuration management (CM) repositories Provides CM guidance and assistance Verifies Version Description Document (VDD) against supplied code

SLM Review Facilitator Facilitates the Test Readiness Review (TRR) and Release Readiness Review (RRR) Prepares and distributes SLM review summary reports

1.6 Test Baselines

All Architecture T&E activity is predicated on the integrity of test baselines established via the placement of all system code, installation instructions (software and database), and the Version Description Document (VDD) in ICE Configuration Management (CM) repositories with versioning and labeling capabilities.

8 OVERVIEW

Test baselines are established in order to provide system managers with a stable point of reference for evaluating whether the system meets its stated requirements. All updates to an established test baseline may only take place with the full knowledge of TA&I and system managers.

If an update to the test baseline is necessary during Architecture T&E, the project team resubmits code, hardware, and/or documentation to CM for placement in the ICE CM repositories. CM then notifies Architecture T&E of the updates to restart independent T&E with appropriate regression testing.

1.7 Architecture T&E Plan Maintenance

ICE follows a continuous improvement approach for the Architecture T&E Plan. E-mail your comments, questions, or suggestions to

ICE-SLMREVIEW@DHS.GOV.

9 DEVELOPMENT TESTING PLAN

2.0 DEVELOPMENT TESTING PLAN

The Development Testing Plan describes how contractor project teams are expected to perform test preparation, execution, and analysis for:

Unit Testing

Integration Testing

Functional Qualification Testing

Development Security Test and Evaluation

Note: The Unit Testing, Integration Testing, and Development Security T&E sections include definitions, key objectives, and process descriptions to provide a complete picture of the testing lifecycle. Plans for Unit Testing and Integration Testing are the responsibility of the project teams.

Contact OISSM for information and guidance concerning Development Security T&E.

2.1 Unit Testing

Unit Testing validates that system units function properly, consistent with system architecture and design. It targets the technical aspects of the code, exercising the specific tasks of the unit while looking for technology and coding issues. Resolution of these issues at this level of Development Testing assists projects in smoothly progressing through the rest of the testing lifecycle and prevents many critical errors from reaching production.

2.1.1 Unit Testing Objectives

Exhibit 9 identifies the key objectives for Unit Testing.

Exhibit 9: Key Unit Test Objectives

Unit Test Focus Objective

Functionality Validate that the specific function of a unit provides the required functionality

Structure and Logic Validate the suitability and accuracy of code execution by examining the logic and algorithms of a unit

Statement Coverage Validate that each statement of the unit functions properly and aligns with system design and architecture

2.1.2 Unit Testing Process Description

During Development, a unit of code (typically the smallest portion of testable functionality) is written and undergoes Unit Testing while isolated from the rest of the system. During Unit Testing, the contractor project team identifies and resolves any issues negatively impacting unit

1 0 DEVELOPMENT TESTING PLAN

functionality and execution. When a unit successfully completes Unit Testing and meets its requirements, the contractor project team can submit the unit for Integration Testing.

2.2 Integration Testing

Integration Testing validates that the interfaces between system units and modules function and communicate properly in combination with each other. It focuses on the interfaces and interaction between units and verifies that each unit continues to provide its functionality as designed while not interfering with the functionality of another unit.

2.2.1 Integration Testing Objectives

Exhibit 10 identifies the key objectives for Integration Testing.

Exhibit 10: Key Integration Test Objectives

Integration Test Focus Objective

Intrasystem Communication

Verify that communication and interaction between system modules function properly by ensuring that calls, communication interfaces, and the integration of commercial off-the-shelf (COTS) software and Government-furnished software (GFS) satisfy system design requirements

Functionality After Integration

Verify that unaltered parts of the application perform as expected when combined with new units (This objective is typically achieved through regression testing.)

2.2.2 Integration Testing Process Description

During integration, units are combined into modules and then into a system. After each unit is integrated, Integration Testing is performed and the contractor project team identifies and resolves any issues negatively impacting module or system functionality and communication.

Integration Testing is complete when all of the units and modules have been successfully integrated into a system and all integration requirements have been met. The integrated system can then be submitted for FQT.

1 1 DEVELOPMENT TESTING PLAN

2.3 Functional Qualification Testing

Functional Qualification Testing (FQT) assesses whether the software product meets the approved requirements described in the SRD.

During FQT, the project team reports problems identified during testing as Test Problem Reports (TPR) in Tracker. Final results are included in the Development Test Analysis Report (TAR), which is submitted to the ICE Electronic Library before the Test Readiness Review (TRR) .

The TRR is conducted to determine whether the system is sufficiently developed, tested, and documented for acceptance into independent testing. During this review, the project team also presents an overview of the test results and provides the test data, test files, or specific system configurations necessary for the implementation of independent testing.

2.3.1 Functional Qualification Testing Strategy

Exhibit 11 identifies and describes testing approaches that can be used by contractor project teams for FQT. (These testing approaches are also performed during SAT in a production-simulated environment. [See Section 3.1])

Exhibit 11: Functional Qualification Testing Approaches

Testing Approach Description

Requirements Testing Tests changes or new functionality against specified requirements (system testing)

User Scenarios Tests the systems based on user-developed test cases or scenarios as compiled from the User Guide or UAT support

Operational Business Process

Tests the system based on actual business processes/workflows that navigate the application

Negative/Error Handling Testing

Tests data/outputs and edits to ensure that the system detects and reports errors properly

Documentation Review Reviews manuals and other related documentation and provides feedback to the project team

Intersystem Testing Tests to ensure that data is correctly passed from system to system

Regression Testing Tests to verify that unaltered parts of the application perform as expected

Multi-platform Testing Tests the application on different platforms or across different sites

2.3.2 Functional Qualification Testing Process

The FQT process can be divided into three sub-processes: Test Preparation, Test Execution, and Test Analysis. Exhibit 12 depicts the FQT process.

1 2 DEVELOPMENT TESTING PLAN

Exhibit 12: Functional Qualification Testing Process Overview

Project team reviews appropriate SLM documentation

(See Exhibit 13)

Contractor project team prepares and submits Development Test Analysis Report

(TAR) to the ICE Electronic Library

Contractor project team conducts functional qualification testing:

Requirements Testing User Scenarios Operational Business Process Negative/Error Handling Testing Documentation Review Intersystem Testing Regression Testing Multi-platform Testing

The contractor project team logs Test Problem Reports (TPR) in Tracker for each defect

Contractor project team collects metrics and reviews results

Contractor Project Team sets up Web and Application servers for

Development Testing

Contractor project team seeds test data as needed

Test Analysis

Test Execution

Test Preparation Development Test Environment

Database Administrators (DBA) set up account and storage

Contractor project team prepares successfully integrated system for Functional

Qualification Testing

Project team presents Functional Qualification Test results at Test Readiness Review

(TRR)

1 3 DEVELOPMENT TESTING PLAN

2.3.2.1 Functional Qualification Test Preparation

Exhibit 13 identifies the key activities performed during the Functional Qualification Test Preparation process.

Exhibit 13: Functional Qualification Test Preparation Activities

Functional Qualification Test Preparation

Inputs Activities Outputs

Integrated System Notice of Intent to Release (NIR) System Change Requests (SCR) System Requirements Document

(SRD)

Requirements Traceability Matrix

(RTM)

Interface Control Agreement

(ICA)

Design Document Data Management Plan (DMP) User Guide Version Description Document

(VDD)

Research the SCRs identified in the release Review SLM documentation Identify system test and environment requirements Develop test cases Contact Data Management to request configuration of development test environment

Development Test Plan Configured Development Test

Environment

2.3.2.2 Functional Qualification Test Execution

Testing is performed in accordance with the Development Test Plan.

Exhibit 14 identifies the key activities performed during the Functional Qualification Test Execution process.

Exhibit 14: Functional Qualification Test Execution Activities

Functional Qualification Test Execution

Inputs Activities Outputs

Development Test Plan Configured Development Test

Environment Integrated System Notice of Intent to Release (NIR) System Change Requests (SCR) System Requirements Document

(SRD)

Requirements Traceability Matrix

(RTM)

Interface Control Agreement (ICA) Design Document Data Management Plan (DMP) User Guide Version Description Document

(VDD)

Execute testing against development test baseline Evaluate results Log Test Problem Reports (TPR) in Tracker for each defect Receive, install, and test any subsequent development test baselines Initiate development of

Development Test Analysis Report (TAR)

Functional Qualification Test Results Test Problem Reports

1 4 DEVELOPMENT TESTING PLAN

2.3.2.3 Functional Qualification Test Analysis

Contractor project teams record the findings in the Development TAR.

The project team submits the report to the ICE Electronic Library and presents the test results to project stakeholders at the TRR. Exhibit 15 identifies the key activities performed during the Functional Qualification Test Analysis process.

Exhibit 15: Functional Qualification Test Analysis Activities

Functional Qualification Test Analysis

Inputs Activities Outputs

Functional Qualification Test Results Test Problem Reports

Analyze Functional Qualification Test results Prepare Development Test

Analysis Report (TAR) Attend Test Readiness Review

(TRR)

Development TAR Updated Version Description

Document (VDD)

2.4 Development Security Test and Evaluation

Development Security Test and Evaluation (ST&E) examines the compliance of the platform, system, or application with the OISSM standard security requirements identified in the SRD and C&A documentation.

2.4.1 Development ST&E Objectives

Exhibit 16 identifies and describes the key objectives for Development ST&E (These testing approaches are also performed during System Security Testing for independent validation of security requirements. [See Section 3.2])

Exhibit 16: Key Development ST&E Objectives

Development ST&E Focus Objective

Web Vulnerability Examines a Web-based application to determine its susceptibility to penetration or misuse

Security Requirements Evaluates system adherence to technical controls within Department security directives and requirements

1 5 DEVELOPMENT TESTING PLAN

2.4.2 Development ST&E Process Description

In preparation for Development ST&E, contractor project teams and OISSM prepare a ST&E Plan that identifies the security testing activities that will be executed. Two high-level activities include:

Establishing an application security baseline

Validating how well a system meets predefined technical control security requirements to prevent unauthorized internal or external access or willful damage

Contractor project teams consolidate the results in the Development ST&E Report, deliver the report to the ICE Electronic Library, and present the results to stakeholders at the TRR.

1 7 FUNCTIONAL TEST AND EVALUATION PLAN

3.0 FUNCTIONAL TEST AND EVALUATION PLAN

The Functional T&E Plan describes how Functional T&E performs system test preparation, execution, and analysis for:

System Acceptance Testing

System Security Testing

User Acceptance Testing Facilitation

3.1 System Acceptance Testing

System Acceptance Testing (SAT) assesses whether the final software product meets the approved requirements described in the existing SRD.

To prepare for SAT, Functional T&E specifies various scope levels for SAT in a Testing Options document. The System Owner, IT Project Manager, and Functional T&E then use this document as a starting point to negotiate what level of testing coverage should be performed. After a testing option is selected, Functional T&E produces an Independent System Test Plan, detailing the activities executed in SAT.

Before SAT begins, a TRR is held to determine whether the system is sufficiently developed, tested, and documented for acceptance into independent testing.

During SAT, testers follow step-by-step manual test procedures and/or execute automated scripts. The results are recorded and compared with expected outcomes. The results are reported in two ways:

Daily status reports are provided to inform stakeholders of any issues or defects discovered during testing.

Problems identified during testing are documented as TPRs in Tracker.

The System Owner and IT Project Manager will review TPRs for disposition. TPRs opened will be traced back to the documented requirements to determine if the system is not operating as designed or if the requirement is incorrect.

If a TPR must be resolved before system release and requires changes to the software, the project team makes the appropriate modifications, tests the fix, and updates the test baseline. The updated system is then delivered to Functional T&E for independent testing. The testing schedule must be flexible enough to handle validation of TPRs and retesting of the system.

At the completion of SAT, Functional T&E summarizes the SAT results in an Independent System Test Analysis Summary (TAS). A Release

1 8 FUNCTIONAL TEST AND EVALUATION PLAN

Readiness Review (RRR) is conducted to determine if and when the release will be implemented into the production environment.

3.1.1 System Acceptance Testing Strategy

Exhibit 17 identifies and describes the various testing approaches used by Functional T&E for SAT. These approaches may be used alone or in combination.

Exhibit 17: System Acceptance Testing Approaches

Testing Approach Description

Requirements Testing Tests changes or new functionality against specified requirements (system testing)

User Scenarios Tests the systems based on user-developed test cases or scenarios as compiled from the User Guide or User Acceptance Testing (UAT) support

Operational Business Process

Tests the system based on actual business processes/workflows that navigate the application

Negative/Error Handling Testing

Tests data/outputs and edits to ensure that the system detects and reports errors properly

Documentation Review Reviews manuals and other related documentation and provides feedback to the project team

Intersystem Testing Tests to ensure that data is correctly passed from system to system

Regression Testing Tests to verify that unaltered parts of the application perform as expected

Multi-platform Testing Tests the application on different platforms or across different sites

3.1.2 System Acceptance Testing Process

The SAT process establishes the methodology for Functional T&E to coordinate, plan, execute, and document the results of the required testing activities during the full lifecycle of development projects. The SAT process comprises three sub-processes: Test Preparation, Test Execution, and Test Analysis. Exhibit 18 depicts the SAT process as performed by Functional T&E.

1 9 FUNCTIONAL TEST AND EVALUATION PLAN

Exhibit 18: System Acceptance Testing Process Overview

Test EnvironmentTest Preparation

Functional T&E develops and distributes Testing Options document

Functional T&E reviews appropriate SLM documentation

(See Exhibit 19)

Functional T&E prepares

Independent System Test Plan

System Owner and IT Project Manager select testing option

Functional T&E prepares

Independent System Test Analysis

Summary (TAS), including Regression Summary attachment for any automated testing results

Functional T&E conducts System Acceptance Testing (SAT):

Requirements Testing User Scenarios Operational Business Process Negative/Error Handling Testing Documentation Review Intersystem Testing Regression Testing Multi-platform Testing

In daily test summaries, Functional T&E informs stakeholders of any issues or defects discovered during testing and logs Test Problem Reports (TPR) in Tracker for each defect

Functional T&E collects metrics and reviews results

Test Lab Engineers and/or Application Integration Services

(AIS) set up Web and Application servers for SAT

Project team certifies lab

Functional T&E, project team, and DBAs seed test data as needed

Test Analysis

Test Execution

Stakeholders attend Test

Readiness Review

Database Administrators (DBA) apply database install packages

Functional T&E delivers TAS at

Release Readiness Review

(RRR)

Configuration Management (CM) ensures that the project team has placed the application code in Version Manager and verifies proper recording of test baseline by project team

2 0 FUNCTIONAL TEST AND EVALUATION PLAN

3.1.2.1 System Acceptance Test Preparation

Exhibit 19 identifies the activities performed during the System Acceptance Test Preparation process.

Exhibit 19: System Acceptance Test Preparation Activities

System Acceptance Test Preparation

Inputs Activities Outputs

Notice of Intent to Release (NIR) System Change Requests (SCR) System Requirements Document

(SRD)

Requirements Traceability Matrix

(RTM)

Interface Control Agreement

(ICA)

Design Document Development Test Plan Data Management Plan (DMP) User Guide Development Test Analysis

Report (TAR) Version Description Document

(VDD)

Research the SCRs identified in the release Review SLM documentation Identify system test and environment requirements Attend training on new business processes Deliver Testing Options document Develop Independent System

Test Plan based on selected testing option for System Acceptance Testing (SAT) Develop test cases Attend Test Readiness Review

(TRR)

Prepare test environment Certify test lab environment

Testing Options document Independent System Test Plan Configured System Test

3.1.2.2 System Acceptance Test Execution

Testing is performed in accordance with the Independent System Test Plan. Exhibit 20 identifies the activities performed during the System Acceptance Test Execution process.

Exhibit 20: System Acceptance Test Execution Activities

System Acceptance Test Execution

Inputs Activities Outputs

Testing Options document Independent System Test Plan Configured System Test

Environment Notice of Intent to Release (NIR) System Change Requests (SCR) System Requirements Document

(SRD)

Requirements Traceability Matrix

(RTM)

Interface Control Agreement (ICA) Design Document Development Test Plan Data Management Plan (DMP) User Guide Development Test Analysis

Report (TAR) Version Description Document

(VDD)

Execute testing against test baseline Evaluate results Log Test Problem Reports (TPR) in Tracker for each defect Inform stakeholders of any issues or defects discovered during testing in daily test summaries Receive, install, and test any subsequent test baselines Attend daily/weekly conference calls as scheduled Initiate development of

Independent System Test Analysis Summary (TAS)

Daily Test Summaries TPRs

2 1 FUNCTIONAL TEST AND EVALUATION PLAN

3.1.2.3 System Acceptance Test Analysis

When testing is complete, Functional T&E prepares the Independent System TAS, reporting the findings. Functional T&E presents the test results to project stakeholders and the contractor project team at the RRR.

Exhibit 21 identifies the activities performed during the System Acceptance Test Analysis process.

Exhibit 21: System Acceptance Test Analysis Activities

System Acceptance Test Analysis

Inputs Activities Outputs

Daily Test Summaries Test Problem Reports (TPR)

Analyze System Acceptance Test results Prepare Independent System

Test Analysis Summary (TAS) Attend Release Readiness

Review (RRR)

Independent System TAS

3.2 System Security Testing

System Security Testing examines the compliance of the platform, system, or application with the OISSM standard security requirements identified in the SRD and C&A documentation. Applications are tested on standard image platform configurations to ensure that normal operations are not impeded by the security configurations themselves.

To prepare for System Security Testing, the project team completes and implements the ST&E Plan and then records the results in a Development ST&E Report during Development Testing.

During independent testing, Functional T&E verifies the results included in the Development ST&E Report by:

Establishing an application security baseline

Validating how well a system meets predefined technical control security requirements to prevent unauthorized internal or external access or willful damage (The OISSM Security Engineer and project team are invited to witness the testing.)

Functional T&E provides the findings and results of System Security Testing to OISSM. OISSM reports any issues that exist to the Information System Security Officer (ISSO) and project team and determines the C&A status of the project and how it should proceed.

3.2.1 System Security Testing Strategy

Exhibit 22 outlines the testing approaches used by Functional T&E during System Security Testing.

2 2 FUNCTIONAL TEST AND EVALUATION PLAN

Exhibit 22: System Security Testing Approaches

Testing Approach Description

Web Vulnerability Assessment

Examines a Web-based application to determine its susceptibility to penetration or misuse

Security Test and Evaluation

Evaluates system adherence to technical controls within Department security directives and requirements

3.2.2 System Security Testing Process

The System Security Testing process establishes the methodology for Functional T&E to execute and document the results of the required testing activities during the full lifecycle of development projects. System Security Testing comprises three distinct sub-processes: Test Preparation, Test Execution, and Test Analysis. Exhibit 23 depicts the System Security Testing process as performed by Functional T&E during Test and Deployment.

Exhibit 23: Test and Deployment System Security Testing Process Overview

Configuration Management (CM) ensures that the project team has placed the application code in Version Manager and verifies proper recording of test baseline by project team

Functional T&E collects metrics and reviews results

The Designated Accrediting

Authority (DAA) determines whether to grant the Authority to Operate (ATO)

The Office of the Information Systems Security Manager (OISSM) receives testing results from Functional T&E and determines how the project team should proceed

Database Administrators (DBA) apply database install packages

Test Environment Test Preparation

Test Execution

Test Analysis

Project team certifies lab

Test Lab Engineers and/or Application Integration Services

(AIS) set up Web and Application servers for System Security Testing

Functional T&E conducts System Security Testing:

Web Vulnerability Assessment Security Test and Evaluation

OISSM presents security test results at Release Readiness Review (RRR) in relation to Certification and Accreditation

(C&A) information

Functional T&E reviews appropriate SLM and security documentation (See Exhibit 25)

Stakeholders attend Test Readiness

Review

(TRR)

2 3 FUNCTIONAL TEST AND EVALUATION PLAN

System Security Testing also occurs during Operations and Maintenance (O&M). Exhibit 24 depicts the System Security Testing process as performed by Functional T&E during O&M.

Exhibit 24: Operations and Maintenance System Security Testing Process Overview

Functional T&E conducts System Security Testing:

Web Vulnerability Assessment Security Test and Evaluation

Functional T&E leverages the production environment

Production Environment

Functional T&E collects metrics and reviews results

The Designated Accrediting

Authority (DAA) determines whether to grant full ATO status

The Office of the Information Systems Security Manager (OISSM) receives testing results from Functional T&E and determines how the project team should proceed

Project team requires full Authority to

Operate (ATO) status

Test Preparation

Test Execution

Test Analysis

3.2.2.1 System Security Test Preparation

Exhibit 25 identifies the activities performed by Functional T&E during the System Security Test Preparation process.

2 4 FUNCTIONAL TEST AND EVALUATION PLAN

Exhibit 25: System Security Test Preparation Activities

System Security Test Preparation

Inputs Activities Outputs

DHS IT Security Program Handbook for 4300A Sensitive Systems Certification and Accreditation

(C&A) documentation Risk Assessment System Security Plan Contingency Plan Security Test and Evaluation

(ST&E) Plan Development ST&E Report System Requirements Document

(SRD)

Version Description Document

(VDD)

System Change Requests (SCR) Notice of Intent to Release (NIR)

Review C&A documentation provided by the contractor project team Review SRD Develop System Security Test scope based on coordination with the Office of the Information Systems Security Manager

(OISSM)

Utilize existing System Acceptance

Testing (SAT) or production environment for security testing, as needed. If not available, configure systems in the Independent Testing Facility according to VDD

Test Scope Configured System Security

Testing Environment

3.2.2.2 System Security Test Execution

System Security Testing is performed in accordance with testing procedures based on OISSM guidance. Exhibit 26 identifies the activities performed by Functional T&E during the System Security Test Execution process.

Exhibit 26: System Security Test Execution Activities

System Security Test Execution

Inputs Activities Outputs

Test Scope Configured System Security

Testing Environment DHS IT Security Program

Handbook for 4300A Sensitive Systems Certification and Accreditation

(C&A) documentation Risk Assessment System Security Plan Contingency Plan Security Test and Evaluation

(T&E) Plan Development ST&E Report System Requirements Document

(SRD)

Version Description Document

(VDD)

System Change Requests (SCR) Notice of Intent to Release (NIR)

Conduct testing Record testing results Deliver results to the Office of the Information Systems Security Manager (OISSM) for inclusion in C&A documentation

Test Results

2 5 FUNCTIONAL TEST AND EVALUATION PLAN

3.2.2.3 System Security Test Analysis

Functional T&E records the results of testing and analyzes the data. Any defects identified are captured as TPRs using Tracker. When testing is complete, Functional T&E prepares a report that documents the results of System Security Testing. Functional T&E then provides the test results to OISSM. Exhibit 27 identifies the activities performed by Functional T&E during the System Security Test Analysis process.

Exhibit 27: System Security Test Analysis Activities

System Security Test Analysis

Inputs Activities Outputs

Test Results Analyze System Security Test results Prepare security test report

System Security Testing report (input for Certification and Accreditation [C&A] Package)

3.3 User Acceptance Testing Facilitation

User Acceptance Testing (UAT) enables actual system users to execute real-world scenarios to test the system before it is deployed. Testing by system users assists the System Owner in determining if the application properly implements business processes and increases user acceptance of new systems.

UAT facilitation is flexible and can be customized to the needs of the users. If needed, support can be provided for developing operational test scenarios, implementing UAT in the Independent Testing Facility, and documenting TPRs during UAT. To reserve the Independent Testing Facility for UAT, the IT Project Manager should submit a request to the ICE Systems Assurance Manager.

To implement a successful UAT, several activities should occur:

Before UAT commences (as time allows), operational test scenarios should be decomposed into actual testing steps, documenting the description and expected result of each test step in the scenario.

Scenarios should…