Attachment J-XX - 2030 Census Program Test and Evaluation Management Plan (TEMP) V2.0 (1).pdf

2030 Census Test and Evaluation

Management Plan (TEMP)

November 25, 2024

Version 2.0 i

Approval Signatures

The 2030 Census Test and Evaluation Management Plan V2.0 has been reviewed and approved for use.

Barbara LoPresti Date Signed Chief, Decennial Information Technology Division

Jennifer Reichert Date Signed Chief, Decennial Census Management Division ii

Document Logs

Sensitivity Assessment

Verification of Document Content

This document does not contain any:

Title 5, Title 13, Title 26, or Title 42 protected information;

Procurement information;

Budgetary information; and/or, Personally identifiable information.

Document Author: Nicole Seamands (DITD) Date: September 17, 2024

Review/Approval

Name Area Represented Date

Daniel Doyle Deputy Chief, Decennial Census Management Division 11/12/2024

Ann Wittenauer Special Assistant, 2030 Census Planning 10/30/2024

Deidre Hicks Chief, Decennial Program Management Office 11/12/2024

Brian De Vos Assistant Division Chief Architecture, Integration, Infrastructure, and Testing 11/12/2024

Version History

Version Date Description

V1.0 08/07/2023 Initial baseline version.

V1.1 07/31/2024 Corrected formatting errors for headings, subheadings, and figures; fixed structural issues to align with 2030 Census IIP, SEMP, and ODMP; ensured the RR section aligns with 2030 Census IIP and added PLTC to the RR section.

V2.0 11/25/2024 Updates from DITD leadership review and comments. Updates to section 3 and 4/4.2 from Yelena Oleynikova comments.

2030 Census Test and Evaluation Management Plan V2.0 iii

Table of Contents

1. Introduction

1.1 Intended Audience

1.2 Document Maintenance

2. Purpose and Scope

2.1 Scope

3. 2030 Census Test Framework

4. Test Levels and Associated Test Efforts

4.1 Test Level: Project

4.1.1 Test Effort: Project Test

4.1.2 Test Effort: Section 508 Test

4.1.3 Test Effort: User Acceptance Test

4.2 Test Level: Program

4.2.1 Test Effort: Program Level Test

4.2.2 Test Effort: Integrated User Acceptance Test (IUAT)

4.2.3 Test Effort: User Reports Test

4.2.4 Test Effort: Output Test

4.3 Test Level: Operational

4.3.1 Test Effort: Operational Readiness Test

4.4 Test Level: Production

4.4.1 Test Effort: Production Test

4.4.2 Soft Launch

4.5 Cross-Level Test Efforts

4.5.1 Cross-Level Test Effort: System Checkout and Certification

4.5.2 Cross-Level Test Effort: Security Controls Test

4.5.3 Cross-Level Test Effort: Performance and Scalability

5. Operations and Maintenance (O&M) Phase Test Activities

6. 2030 Census Environments

7. Test Approach

7.1 Test Planning

7.2 Test Design

7.3 Test Execution

iv

7.4 Test Reporting

7.4.1 Requirements Traceability

7.5 Development, Security, and Operations Testing through CI/CD

7.6 Test Automation

8. Test Data

8.1 Test Data Management

9. Assumptions and Risks

Appendix A: Test Type Detailed Objectives

Appendix B: Test Levels, Efforts, and Types by Owner and Environment

Appendix C: Test Documentation and Document References

List of Tables

Table B-1: Test Levels, Efforts, and Types by Owner and Environment

Table C-2: Test Documentation

List of Figures

Figure 1: 2030 Census Test Framework

Figure 2: Hierarchy of a Notional Test Level with Test Efforts and Test Types

Figure 3: Test Levels

Figure 4: Project Level: Test Efforts and Test Types by Test Owner

Figure 5: Program Level: Test Efforts and Test Types by Test Owner

Figure 6: Operational Level: Test Efforts and Test Types by Test Owner

Figure 7: Production Level: Test Efforts and Test Types by Test Owner

Figure 8: Test Life Cycle

Figure 9: Traceability Relationships and Link Types

Figure 10: DevSecOps Test Management Tool Integration

1. Introduction

The decennial census aims to count the population and housing in the United States and share the results with the President, the states, and the American people. For the 2030 Census, over 335 million people in more than 145 million homes need to be counted. The 2030 Census Program focuses on five main areas to improve the process:

1. Better ways to reach everyone, especially those who are hard-to-count.

2. Update how people in group living situations are counted, such as dorms or nursing homes.

3. Combine data collection and processing to fix issues and improve quality quickly.

4. Simplify support operations to make them more effective.

5. Use new data and methods to improve the final count at the end of the decade.

The 2030 Census Test and Evaluation Management Plan (TEMP) is essential to the success of the 2030 Census Program, providing a structured approach to testing and evaluating systems, methodologies, and operational readiness. By defining the methodologies and performance metrics, the TEMP ensures thorough testing and evaluation throughout the program life cycle, from planning and development to implementation. This systematic approach not only enhances the Census Bureau's ability to identify and mitigate risks but also ensures the 2030 Census Program is conducted with the highest standards of precision and efficiency.

1.1 Intended Audience

The TEMP is intended for those involved in the architecture, engineering, program management, and IT solution development for the 2030 Census:

Decennial Information and Technology Division (DITD).

Decennial Census Management Division (DCMD).

Decennial census purpose-built systems.

Enterprise Initiatives (Business Ecosystem (BE)).

Office of the Chief Information Officer (OCIO).

Decennial Contracts Execution Office (DCEO) (contractor-delivered solutions).

2030 Census key stakeholders such as, Population Division (POP), Decennial Statistical

Studies Division (DSSD), Field Division (FLD), etc.

Third-party contracted solutions.

1.2 Document Maintenance

DITD oversees the creation and upkeep of the TEMP. Regular reviews will be hosted by DITD with relevant stakeholders to gather insights and ensure alignment with the goals of the 2030 Census

Program. These updates are crucial for adapting to new challenges and leveraging technological advancements that will enhance the accuracy and efficiency of the 2030 Census.

2. Purpose and Scope

The TEMP provides essential guidance and structure for testing activities crucial to the success of the 2030 Census Program. This plan aims to:

Ensure comprehensive coverage: Includes necessary test activities to thoroughly evaluate production readiness of the systems and operations.

Standardize procedures: Establish consistent methodologies and practices for conducting tests and incorporating automation to enhance efficiency.

Integrate test efforts: Ensure thorough testing by coordinating and integrating different test efforts and types executed by various test and stakeholder groups.

Optimize resources: Allocate and manage resources efficiently to maximize productivity during testing phases.

Identify and mitigate risks: Conduct early testing and identify potential issues, allowing for early resolution and minimizing issues in production.

Overall, the TEMP serves as the authoritative source for policies, guidance, and best practices governing all testing efforts. It establishes a comprehensive framework and outlines testing activities to ensure thorough validation of the 2030 Census Program.

2.1 Scope

The 2030 Census Program conducts two major field production tests ahead of the 2030 Census: one in 2026 and a dress rehearsal in 2028. The TEMP is written with the 2030 Census in mind, acknowledging that certain deviations may be practical, or necessary, for the 2026 and 2028 field production tests to effectively align with operational requirements and objectives.

Certain test efforts, such as Performance and Scalability, are not included or may have a reduced scope in the 2026 Census Test (26CT). This document will be updated for 2028 Dress Rehearsal (28DR) and 2030 Census to reflect insights from the 26CT and the 28DR, as well as to incorporate any new technological advancements.

3. 2030 Census Test Framework

This document defines the 2030 Census Test Framework, an overall structured approach for verifying and validating that participating systems are designed, built, and implemented to meet the 2030 Census requirements and are operational upon deployment. These groups of functionality are organized into units known as Operational Deliveries (ODs) and the testing of each OD adheres to the Census Test Framework. To identify defects as early as possible and maximize test coverage before the System of Systems (SoS) goes into production and an OD commences operations, the framework encompasses multiple levels tailored to different phases of the 2030 Census Program life cycle. These are further described within Section 4.

Figure 1: 2030 Census Test Framework

4. Test Levels and Associated Test Efforts

As discussed, each test level in the 2030 Census Test Framework consists of test efforts, and test efforts can include a variety of test types. The group with ownership responsibility for a test effort is referred to as the test owner. Although each test effort has an identified test owner, some tests may have test participants from multiple areas or teams in order to achieve the objectives of the test.

The following figure depicts the hierarchy of a notional test effort.

Figure 2: Hierarchy of a Notional Test Level with Test Efforts and Test Types

Notional Test Level

Test Effort 1

Test Type 1

Test Type 2

Test Type 3

Test Effort 2

Test Type 4

Test Type 5

Test Effort 3

Test Type 1

Test Type 6

The 2030 Census Test Framework levels of testing are Project, Program, Operational, and Production. This section will introduce the framework test levels.

Hierarchical Levels of Testing:

Project Level: Focuses on individual components and specific aspects of the system to ensure each system delivers high-quality outputs to satisfy specified requirements.

Program Level: Focuses on coordination and integration of IT solutions and systems to ensure the comprehensive readiness of the SoS.

Operational Level: Focuses on readiness of the Operational Delivery (OD) by ensuring integration of processes, procedures, systems, and people. This enables field and operational staff to conduct user-role-based tests that mirror operational procedures. Thus, ensuring operational readiness by validating procedures, which include monitoring, maintenance, and support processes.

Production Level: Focuses on actual live environment where the system is used by end-users to ensure the system performs correctly in the live environment and meets user expectations.

Figure 3 illustrates various stages of a Program’s life cycle, structured into the four main test levels.

Each level is designed to ensure rigorous evaluation and readiness of the system(s) before progressing to the next level of testing.

At the Project Level, individual applications are developed and tested predominantly through Continuous Integration/Continuous Delivery (CI/CD) processes. The Program Level integrates multiple systems to evaluate interoperability and performance in a scaled environment using simulated data. The transition to the Operational Level focuses on implementing the SoS in a near-production environment to test business process integration, such as procedures and training written for the various solutions. Finally, the Production Level involves testing in the production environment once all required systems for the associated ODs have been tested and approved during operational level testing including verifying that the production environment has been setup successfully. And although not a test level, effort, or type in the framework, Operations and Maintenance (O&M) testing refers to activities involved in verifying and validating the performance, reliability, and stability of a system after it has been deployed and is in operational use. Throughout these test levels, stakeholders evaluate progress and system alignment with business goals at designated evaluation points, maintaining a continuous feedback loop and iterations to enhance system robustness before full-scale operations.

Figure 3: Test Levels

The following subsections provide more detailed explanation of the test efforts and test types performed in each test level. Test efforts that occur across the various levels of the framework are referred to as cross-level test efforts and are described in Section 4.5. Details of all the test types referenced in the framework can be found in Appendix A: Test Type Detailed Objectives.

4.1 Test Level: Project

Project Level testing checks the basic parts of a system to ensure they align with the project’s goals and plans. This testing examines if the main ideas and designs are sound, compares design options, and verifies that requirements are clear and meet stakeholder expectations. Project Level testing is completed, for a set of requirements or newly developed work to ensure that the system’s basic concepts, design, functionality are correct and meet requirements before moving on to more detailed and comprehensive testing.

With Infrastructure as Code (IaC) and a CI/CD pipeline, automation largely handles Project Level testing, continuously validating updates as changes are developed and merged. By embedding these automated tests into the pipeline, teams eliminate the need for environment lockdown, allowing developers to work in parallel while continuously integrating and validating new changes to software.

Figure 4: Project Level: Test Efforts and Test Types by Test Owner

4.1.1 Test Effort: Project Test

Project testing encompasses the rigorous evaluation of software components within the context of a specific development initiative. It aims to validate each individual module or feature meets defined requirements and performs as expected.

4.1.2 Test Effort: Section 508 Test

Section 508, as amended and stipulated in Federal Acquisition Regulation [FAR] 39.201 and Title 36 of the Code of Federal Regulations [C.F.R.] 1194.1, requires federal agencies to develop, procure, maintain, and use Information and Communication Technology (ICT) to ensure:

Individuals with disabilities who are Federal employees have access to and use of information and data that is comparable to the access to and use of the information and data by Federal employees who are not individuals with disabilities.

Individuals with disabilities who are members of the public seeking information or services from a federal department or agency have access to and use of information and data that is comparable to the access to and use of the information and data by such members of the public who are not individuals with disabilities.

Section 508 Certification is expected for all external (public facing) systems and internal systems which have at least 5,000 active users. In all other cases, Section 508 compliancy is expected. The 508 Program Office can share additional information regarding the 508 Certification process.

4.1.3 Test Effort: User Acceptance Test

User Acceptance Testing (UAT) ensures systems meet the business needs of the customer and ultimate end-users. The first iteration of UAT is conducted by stakeholders at the Project Level to support individual system verification of stakeholder and solution requirements as early as possible.

The goal is to ensure the system satisfies the acceptance criteria defined for the systems development.

4.2 Test Level: Program

Program Level testing focuses on the functionality and interactions of the overall SoS. The goal is to validate that the SoS successfully performs all applicable processes outlined in Business Process Models (BPMs) and Workflow Diagrams (WFDs) with data-input to data-output as part of the process. Program Level testing occurs in the Integrated Test Environment (ITE) for the majority of test efforts and in the staging environment for performance and scalability testing (see Section 5 for more details about environments).

Details on Program Level test efforts that occur across the various levels, such as System Checkout and Certification, Performance and Scalability, and Security Controls Testing, are provided in Section

4.5 Cross-Level Testing.

Figure 5: Program Level: Test Efforts and Test Types by Test Owner

4.2.1 Test Effort: Program Level Test

Program Level testing extends beyond individual systems to encompass the integrated functionality of multiple systems within the SoS. It focuses on ensuring these interconnected elements operate harmoniously and fulfill overarching operational objectives.

The Program Level test effort guarantees the reliability, scalability, and interoperability of the SoS and supports assuring that an OD has been fully integrated and tested. This approach ensures successful deployment and operation in real-world environments, mitigation of risks associated with system integration, and delivery of a cohesive, high-quality software solution.

4.2.2 Test Effort: Integrated User Acceptance Test (IUAT)

As introduced earlier, UAT is conducted at the Project Level and ensures individual systems meet the business needs of the customer. At the Program Level, Integrated User Acceptance Testing (IUAT) provides additional opportunities for stakeholders to perform a more enriched acceptance test through integrated, comprehensive, role-based test scenarios that interact with a system’s user interface in the ITE environment.

4.2.3 Test Effort: User Reports Test

User reports testing ensures reports accurately reflect the underlying data and calculations, emphasizing data integrity. The testing involves inputting data into front-end systems, processing it through the relevant workflows, and verifying the output is correctly displayed in the corresponding fields of the generated reports. User reports testing is unique in that it requires upstream data sources and data processing systems to be stable. The objective is to enable users to access required information swiftly and provide reliable insights leading to informed decisions.

4.2.4 Test Effort: Output Test

Output testing validates the systems’ backends are storing, passing, and updating data as expected from beginning to end. Output testing evaluates the preparedness of near real-time and post-collection processing activities prior to the closeout of data collection operations.

4.3 Test Level: Operational

The next test level, Operational, provides stakeholder teams the opportunity to test using user role-based scenarios that integrate operational procedures, training, and system functionality in the Staging environment. Operations which will undergo operational readiness testing are determined by the DCMD Program Architecture and Integration Branch (PAIB). During test planning, PAIB will also outline the roles and responsibilities of supporting team(s) as needed.

Figure 6: Operational Level: Test Efforts and Test Types by Test Owner

4.3.1 Test Effort: Operational Readiness Test

Operational Readiness Test (ORT) contains multiple test types, but one which is unique to this test effort is the Operational Process Test. The Operational Process Test type involves verifying that users across different operational roles can perform all their necessary activities seamlessly within the integrated SoS. In addition, it involves validating operational procedures, including monitoring, maintenance, and support processes, to ensure they are ready for use by the operation. One of the outcomes of the test includes reviewing and validating operational documentation such as process guidelines and procedure manuals to ensure accuracy, comprehensiveness, and accessibility.

4.4 Test Level: Production

Production Level testing is performed in the production environment once all required systems for the associated ODs have been tested and approved during Operational Level testing.

Figure 7: Production Level: Test Efforts and Test Types by Test Owner

4.4.1 Test Effort: Production Test

Production test provides a final opportunity to ensure that the systems is ready for use. These tests are crucial to verify the systems function correctly, perform as expected, and are ready for the operation to commence.

4.4.2 Soft Launch

Although not a specific test effort or a type of test, some operations include a Soft Launch in this level of the framework. Soft Launch is an early run-through of the production operation to ensure all functional and operational components, as well as data, flows through all the systems and performs as expected in advance of a full launch. Soft launch is conducted with a subset of users (office staff, field staff, HQ staff, etc.), and is inclusive of all operational activities relative to the OD (such as hiring, account management, training, etc.). It should be noted that data resulting from Soft Launch activity in Production is used in the creation of data products as Soft Launch is not considered a test.

4.5 Cross-Level Test Efforts

As previously mentioned, cross-level test efforts refer to test efforts that span multiple test levels of the framework and can occur in any combination of the project, program, operational, and production test levels. This section will explore cross-level test efforts that may start as early as development and continue through production.

4.5.1 Cross-Level Test Effort: System Checkout and Certification

System Checkout and Certification (C&C) test effort is an engineering checkout which confirms system infrastructure has been installed correctly and is conducted in all environments as needed.

System C&C occurs following infrastructure handover and as systems go through configuration changes and/or updates. Changes may include infrastructure migrations (standardization, consolidation, application, data center, or database migrations) or other modifications to underlying information technology (IT) infrastructure including environment provisioning, patching and operating system (OS) upgrades, database updates, etc. System C&C is led by the infrastructure team and supported by other test areas, such as the Project Level teams and the 2030 Census Program Level Test (PLT) Team. The implementation of IaC practices does impact the level of effort and frequency that teams put forth in support of System C&C. IaC automates the creation, configuration, and deployment of infrastructure components such as servers, networks, storage, etc., enabling infrastructure to be more reliable and consistent across environments. Though IaC does not eliminate the need for infrastructure testing altogether, it does exponentially reduce the time spent performing the required validations by providing teams with a standardized and automated method to conduct validations thoroughly.

4.5.2 Cross-Level Test Effort: Security Controls Test

The Security Controls Test is essential for ensuring system resilience and compliance across the Census Bureau. Aligned with federal guidelines from entities like the National Institute of Standards and Technology (NIST), Department of Commerce, and Census Bureau Security Standards, this approach emphasizes early security measures to effectively mitigate vulnerabilities and reduce incident recovery times. Technical controls are implemented based on NIST system security categorizations, leveraging existing Census Bureau-wide security services to maintain consistency and effectiveness across all environments.

The security testing framework encompasses several critical objectives. It includes early identification of vulnerabilities and validation of implemented security controls to ensure their intended functionality. This rigorous testing regime not only mitigates risks, but also ensures compliance with regulatory requirements. System integrity is certified thorough testing post-installation and after any configuration changes or updates, ensuring adherence to acceptance criteria.

Throughout the development process, Security Controls Test follows a Development, Security, and Operations (DevSecOps) methodology. This approach integrates automated security tests such as Static Application Security Testing (SAST), Dynamic Application Security Testing (DAST), Runtime Application Self-Protection (RASP), Interactive Application Security Testing (IAST), Dependency Scanning, Container Scanning, and License Scanning into the CI/CD pipeline, enabling early detection and resolution of security issues. Collaboration among infrastructure teams, middleware teams, project development teams, and the security team ensures that security remains a top priority starting with initial development and continuing throughout the life cycle.

Additionally, the strategy includes ongoing penetration testing and advanced persistent threat emulation. These measures enable continuous assessment and strengthening of defenses against emerging threats, ensuring robust system security capable of withstanding evolving cybersecurity challenges.

4.5.3 Cross-Level Test Effort: Performance and Scalability

Performance and Scalability (P&S) tests are critical evaluations conducted to ensure software applications perform well under expected loads and can handle increased demands. P&S testing for the 2030 Census Program involves analyzing workload demand models and system architecture to develop system models, establish baselines, and define performance metrics for each system. The objective of this effort is to:

Assess how responsive, fast, and stable a system is under various workloads.

Uncover and resolve performance bottlenecks before deployment to ensure the application meets performance expectations.

Examine how well a system can grow or handle increased load without sacrificing performance.

Evaluate a system's ability to scale up (handle more load) or scale out (handle more users/transactions) effectively.

Refer to the Performance and Scalability Test Plan for additional details on performance testing at each test level. This test effort may have a reduced scope or not be included in the 26CT.

5. Operations and Maintenance (O&M) Phase Test Activities

O&M, though not a test level, effort, or type in the framework, is a critical phase that introduces a different scope of testing. O&M testing refers to the activities involved in verifying and validating the performance, reliability, and stability of a system after it has been deployed and is in operational use. Testing activities in this Systems Development Life Cycle (SDLC) phase ensure that the system continues to function as expected and meets the necessary operational standards throughout its life cycle.

Changes to software and systems, especially after the initial release to production, are almost inevitable. O&M testing is performed to evaluate the success of software and system changes, and to check for possible unintended side effects in unchanged parts of the system through regression testing. Testing during O&M is conducted for planned and unplanned releases and is triggered by:

1. Modifications – Planned enhancements, corrective, and emergency changes of operational functionality or the environment (database upgrades, software patches, Commercial-off-the- Shelf (COTS) upgrades, etc.).

2. Migrations – An application or software migrates from one platform to another and includes testing of the new environment as well as the changed software or converted data.

3. Retirement – When an application reaches the end of its operational life cycle and can be retired, this includes testing of data migration/archiving, and decommissioning.

The IT Production Operations Management Plan outlines the process for managing 2030 Census production operations. All systems, stakeholders, and service providers use the IT Production Operations Management Plan to understand the expectations for production OD support, incident reporting and escalation, rapid response processes, and ODs integration with Enterprise O&M processes.

6. 2030 Census Environments

In Section 4, the various test levels and their associated test efforts were discussed, and depending on the SDLC phase, the test level, effort, and type, multiple different environments are utilized to fulfill the testing needs. This section will describe the census environments planned to be used and the various test efforts they support. It should be noted that other programs and areas that the 2030 Census Program interacts with, such as the Business Ecosystem, may have a different set of environments with different names and usage than what is described in this section.

In software development, an environment refers to a specific configuration of hardware and software that hosts an application or system. It provides a controlled setting where various aspects of the software can be tested, evaluated, and deployed. Production is the live environment where the finalized software is deployed and used by end-users – representing the actual operational environment. Test environments are essential for mitigating risks, ensuring quality, and identifying potential issues before software deploys into production. Below are the environments used by the 2030 Census Program:

Development (DEV): An isolated environment that allows developers to create, modify, and test code at their own discretion.

Quality Assurance (QA): An environment where Project Level testers evaluate the functionality and usability of the code output from the DEV environment.

Integrated Test Environment (ITE): An environment where testing teams perform system integration ensuring end-to-end functionality.

Staging (STG): An environment that mimics the production environment where systems are scaled for higher capacity than lower environments.

Production (PROD): The live environment where software is deployed and used by end-users.

Appendix B: Test Levels, Efforts, and Types by Owner and Environment provides a table indicating the test levels, efforts, and types conducted in the various census environments.

7. Test Approach

At all test levels, the test approach begins with meticulous planning, leveraging system specifications, requirements, and workflow diagrams as the foundation. This planning process is enriched by insights gathered from stakeholders and the OD team to ensure plans align with the intended system functionality and operational needs. Clear and continuous communication of progress, issues, and risks occurs through various channels, ensuring project teams and stakeholders are consistently updated on the status of testing efforts.

Figure 8 depicts the cyclical-phased test approach employed by the 2030 Census test teams. Due to the iterative nature of testing, different 2030 Census test teams may be engaged in multiple phases of the test life cycle simultaneously.

The implementation of CI/CD practices facilitates ongoing test design, execution, and reporting as new automation tests are seamlessly integrated into the CI/CD pipeline. This approach ensures testing remains integrated with development activities, promotes a “shift-left” testing mindset, and allows defects to be identified and resolved earlier in the software development life cycle.

Figure 8: Test Life Cycle

7.1 Test Planning

During the Test Planning phase, the 2030 Census test team develops an approach to break down procedures into test scenarios and further into test cases, categorized by test types. The 2030 Census test teams update these based on various inputs, including system documents, stakeholder and solution requirements, architecture and system design models, workflow diagrams, business threads, and existing test cases.

The 2030 Census test teams specify these inputs and outline when they will be able to deliver test outcomes to a respective OD team. They also identify dependencies, such as system access and required documentation, to determine the impact to test plans and work towards resolving any issues from a testing perspective. Test planning is an agile and ongoing process. As requirements change, the 2030 Census test teams update plans and scenarios as applicable.

7.2 Test Design

In the Test Design phase, the 2030 Census test teams identify how testing is to be conducted, which involves identifying test scenarios, detailed test cases to verify specific requirements using operational scenarios, required test data, and expected test results. They also define acceptance criteria in collaboration with project teams and business Subject Matter Experts (SMEs) as applicable. This iterative process involves adding test procedures, scenarios, test cases, and test steps as necessary to ensure comprehensive test coverage for the functionality.

7.3 Test Execution

In the Test Execution phase, the approach is to test every system and integration point as early as possible using various test types. Automated test execution within the CI/CD pipeline is the highest priority. Manual test execution is prioritized based on the criticality of the requirements, such as:

Stakeholder and solution requirements (features/user stories) for Project Level test efforts.

Stakeholder requirements and business threads for Program Level test efforts.

Demand models for performance and scalability tests.

Throughout testing, the 2030 Census test teams record and communicate observations and defects.

Both Project Level and Program Level testing teams use a defect management tool for tracking defects. Detailed information regarding defect identification and resolution processes for DITD are described in the DITD PLT Defect Management Plan (DMP).

7.4 Test Reporting

The 2030 Census program management relies on status reports and metrics to measure progress.

Test metrics are collected and reported at all test levels and serve as key inputs for readiness review gates. The 2030 Census test teams provide summary reports to 2030 Census Program leadership, including the following metrics where appropriate:

1. Execution Metrics:

Status on test execution results, used to determine readiness of components and systems through project, program, and operational test levels.

The various 2030 Census test teams deliver Test Analysis Reports (TARs) for each OD prior to Production Readiness Review (PRR). TARs are made available to the system teams, OD Team, and the 2030 Census Program leadership.

2. Defect Metrics:

The PLT DMP defines what constitutes observations and defects and provides guidance on assigning severity and priority values to them.

Status of observations and defects includes all resolved and active defects identified during testing.

Metrics include the total number of defects categorized by system, status, severity, test type, and other relevant information, as well as defect aging and density.

3. Coverage and Quality Metrics:

Monitoring and measuring the quality of test activities involves several key metrics. These metrics provide insights into the effectiveness and thoroughness of testing efforts, aiding in the improvement of test suites, and enabling stakeholders to understand the extent of testing relative to requirements. Key metrics include:

Code Coverage: Measures the percentage of code executed by tests. This helps identify areas of code that are not adequately covered by tests.

Requirements Coverage: Evaluates the proportion of requirements that have corresponding test cases ensuring that all specified functionality is tested. Requirements traceability is discussed in greater depth in Section 7.4.1.

Paths Coverage: Determines the percentage of executable paths through the code that have been tested.

Branch Coverage: Measures the proportion of decision points (branches) in the code that have been executed by tests.

Regression Coverage: Frequency and volume of regression checks as products grow in function and complexity.

Data-Oriented Coverage: Analysis of input and output parameters to ensure test scenarios cover functional and data perspectives.

Automation Coverage: Percentage of test coverage achieved through automation testing.

These metrics provide a comprehensive view of testing progress and quality, ensuring that all aspects of the 2030 Census systems are thoroughly evaluated and ready for operational use.

7.4.1 Requirements Traceability

The 2030 Census test teams ensure comprehensive test coverage and traceability of requirements using a Requirements Traceability Verification Matrix (RTVM). The RTVM links each requirement to its corresponding verification tests within business threads, enabling streamlined, transparent, and automated status reporting for 2030 Census Program leadership and stakeholders.

The 2030 Census IT test management tool suite consists of the Census Bureau enterprise project management tool (Jira), and test management tool (Xray) to link requirements to business threads, which are linked to test cases, however 2030 Census test teams outside of DITD may have their own test management tools they utilize.

Requirement verification statuses update in real-time based on the linked test execution statuses.

This methodology will ensure reporting to stakeholders is done efficiently and transparently. Figure 9 illustrates the relationships and link types of stakeholder requirements, business threads, defects, and test issue types.

Figure 9: Traceability Relationships and Link Types

7.5 Development, Security, and Operations Testing through CI/CD

Project Teams delivering systems to the 2030 Census Program follow a CI/CD practice. This practice uses DevSecOps orchestration, automation, tooling, and governance processes to eliminate silos between development, testing, security, and operations teams. The adoption of CI/CD offers several benefits, including improved agility, reliability, and efficiency in both system development and testing processes. Given the risk of compressed timelines and frequent software changes, the repeatable processes provided by a DevSecOps CI/CD practice enhance confidence in the delivered systems and reduce program risk.

The DITD DevSecOps team is responsible for supporting the integration of tests within automated pipelines, while the 2030 Census test teams are responsible for the development and continued maintenance of test automation scripts. The goal is to create a practical test automation framework for 2030 Census systems that integrates seamlessly with the CI/CD pipeline. The program maximizes the use of automated testing to improve efficiency and quality, aiming to simplify as much of the testing as possible. Therefore, test environments are set up with test automation as a foundational capability.

Automated security checks and custom test scripts (including, but not limited to, infrastructure, unit, component, functional, interface, and system integration tests) are incorporated into the release cycle. This ensures that code changes and application configurations are validated before each release, reducing delivery time. Additionally, DevSecOps tools automatically generate and publish test result reports to facilitate systematic approvals of releases. For more information on the

DevSecOps CI/CD implementation strategy, review the DITD DevSecOps Reference Design.1 Figure 10 depicts the integration of DevSecOps with the test management tool.

Figure 10: DevSecOps Test Management Tool Integration

7.6 Test Automation

Test automation is the process of using software scripts and tools to execute and validate tests automatically. It involves using scripts to control the implementation of tests, compare actual results with anticipated results, and report the results of the tests. By automating tests, 2030 Census test teams accelerate their testing cycles, broaden the scope of what can be tested quickly, and facilitate more frequent testing. Ultimately, the primary objective of test automation is to streamline the testing process, reducing human effort and time while enhancing the accuracy and consistency of testing efforts.

The industry shift toward automated testing has been unfolding for many years, but test automation supports modern software development in ways that go beyond just delivering better reliability, speed, and efficiency. For example, automated tests are incorporated within CI/CD pipelines; this feature also known as “shifting left,” allows project and program test participants to perform End-to-End Integration (EEI) tests earlier and more frequently within the SDLC. For additional details on the development of an automation framework, processes for tool selection, system evaluation, and automation implementation refer to the Automation Test Plan (ATP).Error!

Reference source not found.

8. Test Data

Test data creation and maintenance are critical aspects of successful testing within the 2030 Census ecosystem. 2030 Census test teams manage different types of test data:

1 DITD DevSecOps Reference Design.docx.

Foundational Data: This type of data can be referred to as "source data” and serves as the basis for initializing most test efforts. It includes essential elements such as universe data, necessary for initiating data generation processes. Details of the management strategy can be found in Section 8.1.

Input Data: This type of data can be categorized as any user-generated data; data which is created through system interaction. Input data, often relies on foundational data being in place to accurately simulate real-world scenarios or specific test conditions.

8.1 Test Data Management

2030 Census test teams each oversee their own input data life cycle management, including defining requirements, generating data, and the ongoing maintenance. On the other hand, the 2030 Census PLT Team assumes sole responsibility of foundational data for all test efforts in the ITE and STG environments.

Foundational data can either be existing title data or synthetic. If a title dataset(s) (Title 5, Title 13, or Title 26) is available and approved, the 2030 Census PLT Team expects to receive these datasets from the respective owners and is responsible for integrating the data within the environment(s).

Otherwise, the 2030 Census PLT Team uses automated processes to generate and maintain synthetic data. Data used to develop and generate synthetic dataset(s) should be created in accordance with industry best practices and should be absent of any data that is founded in existing titled dataset(s).

DITD has adopted a best practice policy of only utilizing “non-title” test data in the “lower” census environments of DEV and QA. Title test data may be utilized in the “higher” census environments of ITE and STG where greater security control restrictions are in place and when appropriate ATO approvals have been obtained. If a need arises for title test data in DEV or QA, then a waiver through the DITD waiver request process can be requested which indicated the purpose, rationale, sensitive datasets to be used, duration of usage, and the security controls to be used to protect the sensitive data.

The test data management strategy involves systematically sourcing production-like data and constructing in-house synthetic sample data in a scalable, reusable fashion. These sample datasets are hosted in a data warehouse, organized into schemas to ensure integrity of the data. Within the warehouse, data is further analyzed and tailored to ensure geographic, personal, and household attributes are in accordance with expected, realistic conditions.

Refer to the Test Data Management (TDM) Plan for additional details regarding the process and implementation of test data generation and maintenance of foundational datasets.

9. Assumptions and Risks

Assumptions and risks are used to plan the test strategy, Program Level test schedules, and Program Level test plans. As assumptions and constraints change, impacts on the planning and approach of the 2030 Census test teams may occur. In these cases, Change Requests (CRs) will be submitted and reviewed to update plans accordingly. The following assumptions are noted:

With respect to all test initiatives:

Requirements allocated to an OD are reviewed and categorized as “Testable” or “Not Testable”.

Participating Enterprise systems (e.g., Data Ingest and Collection for the Enterprise (DICE) and Enterprise Data Lake (EDL)) must be engaged throughout the entire process to ensure adherence to designated tasks and timelines.

Environments and systems planning to use title data must acquire Authorization to Operate

(ATO).

Personnel interacting with title data are trained and cleared to work with title data.

With respect to Program Level test efforts:

The 2030 Census PLT Team is the owner of the ITE and STG environments and will coordinate with other 2030 Census test teams on their test activities.

PLT efforts are conducted within the ITE and/or STG environment, which shall mirror the Production environment.

Mobile applications are installed on devices through the Mobile Device Management (MDM) solution.

Mobile devices are provided and provisioned to the 2030 Census PLT Team for testing purposes by the MDM.

The MDM team maintains record of all devices provided to the 2030 Census test teams.

Mobile device requirements and footprints are baselined before Program Test Initialization

Checkpoint (PTIC).

The following are risks:

All systems supporting 2030 Census may not have an ITE required to conduct integration testing. In these cases, the 2030 Census PLT Team and system owners collaborate to identify solutions or gaps in test coverage.

Environments not stood up in time.

Environments not mirroring production to the extent needed.

Given interaction with various Census Bureau programs, decennial census testing incurs risks to schedule coordination, environment consistency, environment and/or system access, and data consistency with partner programs.

Delayed code delivery.

Lack of documentation.

Lack of sufficient test data (or enough data to scale).

Appendix A: Test Type Detailed Objectives

Test efforts from Project Level through Production Level including cross-level test efforts may perform the following test types where appropriate.

Unit Test o Unit testing isolates the smallest piece of testable software from the remainder of the code and determines whether it behaves as intended.

o Unit tests reduce difficulties of discovering errors contained in more complex pieces of the application, enhance test coverage, and increase confidence in changing/maintaining code.

o Tests individual software units to verify code correctness and functionality.

Commonly uses white box testing.

Component Test o Component testing (also referred to as module testing) validates each individual component without integrating with other components.

o Unlike unit testing (which appraises the system at a granular level), component testing assesses the integration of units at an application level by executing use cases.

o Component tests ensure it meets its functional and non-functional requirements and minimize the risk of introducing defects when integrating with other components.

Pairwise Test o Pairwise testing focuses on validating the integration of nearest-neighbor systems to verify that functional, technical, and solution requirements, as well as quality of standards, are met before their systems enter independent Program Level testing.

o Tests data flow paths, inputs, outputs, and interdependencies between systems.

Focuses on one system and its nearest neighbors, using a cost-effective method to find errors early.

Application Programming Interfaces (API)/Component Base Performance Test o Assesses the initial performance of an application before any code is written, as well as while code is in its early stages. This testing focuses on performance of both application hardware technology (networks, load balancers, database, and web servers, etc.) and software.

o By testing the base technology early, project development teams can better optimize components of the system and avert business costs later.

Functional Test o Ensures each system function conforms to specifications. Includes:

Mainline Functions: Ensures basic functional requirements are met.

Negative Testing: Focuses on ensuring the system can handle a wide range of invalid inputs and unexpected situations without crashing.

Error Handling: Tests specific scenarios where errors are expected to ensure appropriate responses and error notifications are triggered.

Regression Testing o Regression tests are a type of testing that aims to ensure that recent code changes or enhancements have not adversely affected existing features or functionalities of a software application.

o Focus on uncovering defects or unintended consequences introduced by changes in the software, and therefore the scope of regression testing may vary depending on the impact of changes.

o Regression testing is crucial for maintaining software quality and reducing the risk of releasing faulty software to users.

Infrastructure Test o Infrastructure tests refer to specific testing focused on validating the underlying technical components and infrastructure that support the system's operation. The validation may include:

Network Components: Testing network configurations, protocols, and bandwidth to ensure they meet the system's requirements for communication and data transfer.

Hardware Components: Verifying the functionality and performance of servers, storage devices, routers, and other hardware elements critical to system operation.

Software Dependencies: Testing compatibility and integration with operating systems, middleware, databases, and other software components essential for the system’s functionality.

System Integration Testing (SIT) o SIT is the process of combining the various systems into one overall program.

o The focus of SIT is executing tests across system boundaries, validating the physical and logical interfaces between systems, hardware products, software products, and external interfaces.

o SIT also ensures that data flows correctly between systems and that integrated components function together as expected.

End-to-End Integration (EEI) Testing o Confirms that an application operates and performs as designed from start to finish, ensuring data integrity across the SoS.

o EEI testing is conducted when software modules or systems are combined and tested as a group across the entire program after the Test Readiness Review (TRR) milestone is achieved. EEI verifies the functional and integration solution requirements for key design items after each component has been integrated at the Program Level and run through the operations during test as intended during…