Website Testing and Quality Assurance Requirements

Contents

Purpose Defect Review Unit Testing Integration Testing System Testing Readiness Review Pre-Launch Production Testing Post-Launch Production Verification Post-Production Monitoring Period Exit from Post-Production Monitoring Period and Acceptance Re-invocation of Post-Production Monitoring Period Website Exception Period Helpful Resources

Purpose

The Website Testing and Quality Assurance Requirements document serves as a guide to ensure the specified website requirements and standards are met (see Section C.3.5). The Contractor shall encourage the Office to actively participate in all phases of testing.

Contractors shall refer to the internal SAA knowledgebase site for additional guidelines and requirements.

Defect Review

The Contractor shall implement a defect assessment process, adhering to the Defect Remediation Protocol (see Section C.3.8), during all phases of Testing to identify and prevent defects.

At a minimum, the Contractor shall classify defects as one of the following:

• Critical – impacts the functionality of the website beyond the specific function causing the defect, with no workaround;

• Serious, with workaround – required functionality cannot be performed, but there are other alternative methods an appropriate user can use to achieve the desired functionality; or,

• Other – any other type of defect (e.g., cosmetic anomalies, misspellings, etc.).

Output: The Contractor shall document not only the cause of the defect(s), but also the proposed solution(s). The Contractor shall submit this information to the Office POC (and copy the COR), who must approve the proposed solution(s) in writing, before proceeding to install the remediation(s) of the defect(s).

Unit Testing

Where applicable, the Contractor shall conduct Unit Tests to evaluate whether the necessary website components have been installed correctly and are functioning properly.

Output: The Contractor shall keep a record of all unit tests, and these tests shall be made accessible for Office or SAA review, upon request.

Integration Testing

The Contractor shall develop and perform a set of Integration Tests. The purpose of the Integration Tests is to ensure that all the website’s various components function together successfully after they have been installed and are operational.

The Integration Tests shall evaluate, at a minimum:

• the accuracy of the data output from form(s) processing;

• the successful delivery of forms-based email to its intended destination;

• verify that web forms only interact with approved constituent correspondence services;

• the accuracy and reliability of searches;

• the accuracy and reliability of links;

• the behavior of any graphics;

• the website’s correct interaction with any necessary external components (e.g., databases, systems, browsers, screens, etc.); and,

• the accuracy and completeness of Content Migration (if any) performed by the SAA;

• that only approved plugins are used to achieve the desired outcome; and,

• the website and its components are in compliance with SAA software policy

Output: The Contractor shall keep a record of all integration tests, and these tests shall be made accessible for Office or SAA review, upon request.

System Testing

The Contractor shall develop and perform a System Testing. The purpose of System Testing is to ensure the entire website is working properly.

The System Tests shall evaluate, at a minimum:

• that the website meets the requirements as specified (see Section C.3.4), and in the individual TOPR; and,

• that the website otherwise functions as expected.

Output 1: Upon completion of the System tests, the Contractor shall report and review all results with the Office POC and submit those items that require SAA assistance for resolution to the SAA.

Output 2: The Contractor shall prepare and present to the Office POC a report (and copy the COR) of the System Tests execution results, and successful remediation of all defects. Upon Office POC’s written approval of those results, to include successful end-user review/testing, the system shall be considered ready for the Readiness Review.

Readiness Review

In preparation for moving the website into production and prior to the Senate Office’s acceptance of the final website, the Contractor shall perform a Readiness Review to ensure that any and all previously identified website deficiencies and/or anomalies have been rectified, and that all the Office’s concerns have been addressed and remedied.

The Readiness Review shall also, at a minimum:

• confirm that Content Migration (if any) has been successfully completed;

• confirm the website functions properly with a fully loaded and current database;

• confirm the successful correction of all defects identified;

• confirm that the website is operating correctly on the test environment authoring backend servers;

• confirm that the Senate Office has been successfully trained to maintain their website (see Section

C.3.9);

• confirm that documentation has been provided to the Senate Office (see Section C.3.9);

• confirm all web forms function properly; and,

• confirm all guidelines set forth by the SAA have been met (refer to the internal SAA knowledgebase site).

Output: Following the successful completion of all facets of the Readiness Review, the Contractor shall submit a Readiness Review Report to the Office POC (and copy the COR). Upon Office POC’s approval of the Readiness Review Report, the Contractor shall coordinate with the Senate Office and the SAA to move the website from the test environment to the production environment.

Once the Readiness Review is complete, the website shall be considered ready to move to production.

When scheduling a website’s move to production, the Contractor shall comply with the website launch schedule set forth by the SAA. The Contractor must ensure that all support resources are available to quickly resolve any post-production problems.

Pre-Launch Production Testing

Following the website’s move to the production environment, but prior to the website being added to the external Domain Name System (DNS), the Contractor shall test the website and confirm, at a minimum, that:

• all website component configurations and their interactions with external components, permissions, and links are working correctly in the production environment public-facing frontend servers, and backend servers; and,

• the website is fully functional and works as designed.

The Contractor shall take into consideration that addressing will not permit two production websites for a single Office to exist simultaneously on the same set of servers. As a result, when necessary, the Contractor shall be prepared to allocate additional time and staff resources during testing to expedite problem identification and resolution.

Output: The Contractor shall document and report problems, defects, anomalies, and proposed solutions to the Office POC (and copy the COR). Upon the Office POC’s written approval, the Contractor will apply the remediated code to the testing tier, and re-test the website. If no additional problems are noted, the website can be moved to the production environment once again and shall then be considered ready for Post-Launch Production Verification.

If no problems are detected, the Contractor shall provide written notification to the Office POC that the website has been successfully tested and is now ready to go live.

Post-Launch Production Verification

Once Website and Quality Assurance Testing has been successfully completed and the website has been added to the external DNS, the Contractor shall confirm that the website is fully operational, working as designed as well as accessible to and completely functional for external visitors.

The Contractor shall comply with production requirements set forth by the SAA.

Output: The Contractor shall notify the Office POC (and copy the COR) in writing when the website goes live, indicating that the Post-Launch Production Verification has been performed, and that the Post- Production Monitoring Period has started.

Post-Production Monitoring Period

During this period, the Contractor shall monitor the website to ensure that, at a minimum, the website:

• properly handles a wide range of input conditions;

• functions properly under heavy load conditions (i.e., performance testing);

• meets stated usability and accessibility criteria;

• is tolerant of normal fault conditions; and,

• behaves identically for users on the Senate network and those on the Internet.

Exit from Post-Production Monitoring Period and Acceptance

The Post-Production Monitoring Period shall end when all defects identified during the Post- Production Monitoring Period have been remediated and the website has performed as required for a period of fourteen (14) consecutive days.

Output 1: In accordance with the Acceptance provisions in Section E.4, at the end of the Post-Production Monitoring Period, the Contractor shall promptly prepare and present to the Office POC a report of the website’s behavior during the Post-Production Monitoring Period, noting any failures, deficiencies, defects and/or anomalies (Website Failure Downtime). If the Monitoring Period has to be restarted (see Re-invocation of Post-Production Monitoring Period below), then the Contractor shall report only the results of the latest Monitoring Period.

Output 2: In accordance with the provisions in Section E.4, if there are no outstanding failures, deficiencies, defects or anomalies, the Contractor may present the website to the Office POC for written Acceptance, which shall constitute the Office’s final acceptance of the website as an operational, production website, as specified in the Task Order.

Re-invocation of Post-Production Monitoring Period

If Website Failure Downtime occurs, the Post-Production Monitoring Period shall be re-started, and shall continue until the standard of performance has been achieved (fourteen (14) consecutive days without Failure Downtime) and all defects identified during the Post-Production Monitoring Period have been remediated.

If a Website Failure Downtime occurrence or website defect causes the Senate Office to fall back to the old website, the Contractor shall, at its own expense, re-convert any new and updated data created in the old website to the new website prior to re-establishing the new website for continued Post-Production Monitoring.

Output: The Post-Production Monitoring Period shall begin again after the replacement of the designated parts, restoration of data from the old website, and notice from Contractor to the Senate Office POC, in writing, (with a copy sent to the COR) that the website is again ready for the Post- Production Monitoring Period.

Website Exception Period

Following the Senate Office’s Acceptance of the website, there shall be an Exception Period of thirty (30) calendar days during which the Contractor shall provide Website Maintenance and End-user Support at no cost to the Office.

Output 1: The Contractor shall notify the Office in writing (and copy the COR) when the Website Exception Period has begun, when the Website Exception Period ends, and confirm in writing that Website Maintenance and End-user Support shall be provided at no cost to the Office.

Output 2: The Contractor shall also present to the Office POC and the COR, a single electronic report of all required documentation, for the record.

The Exception Period does not include the development of new capabilities or new content items.

Charges for Website Maintenance and End-user Support (when purchased by the Senate Office) shall begin the day after the last day of the Exception Period.

Helpful Resources

Section 508 Accessibility News: The Section 508 Update | Section508.gov W3C Accessibility Standards Overview | Web Accessibility Initiative (WAI) | W3C Digital Experience Usability – Digital.gov Usability Starter Kit – Digital.gov Helpful Checklists U.S. Senate: Internet Services and Technology Resources Usage Rules https://www.section508.gov/ https://www.section508.gov/blog/accessibility-news-the-section-508-Update/ https://www.w3.org/WAI/standards-guidelines/ https://www.gsa.gov/technology/government-it-initiatives/digital-experience https://digital.gov/topics/usability/ https://digital.gov/resources/digitalgov-user-experience-resources/digitalgov-user-experience-program-usability-starter-kit https://digital.gov/resources/checklist-of-requirements-for-federal-digital-services/ https://www.senate.gov/usage/internetpolicy.htm