PWS Attachment 12_Quality Management Guide v1.05.pdf

Rural Development

Quality Management Guide

Version 1.05

Last Updated: 07/19/2021 Point of Contact:

Ryan Schneider Test Architect

Approved by:

Therese Gallagher, RDTO CISO

Rural Development 03/10/2021 Quality Management Guide (QMG)

Page | i

REVIEW & CHANGE HISTORY

Version Date Name Change History

1.0 08/21/2020 Ryan Schneider First Draft

1.01 12/08/2020 Sarah Hallet- Reeves

Reviewed and provided Edits

1.02 01/07/2021 Ryan Schneider Updated and solidified edits

1.03 02/26/2021 Ryan Schneider Updated and solidified edits

1.04 03/10/2021 Ryan Schneider Final Draft

1.05 07/19/2021 Ryan Schneider Updated per PMO Review

Page | ii

Table of Contents

1 QUALITY MANAGEMENT PROCESS

1.1 PURPOSE OF QUALITY MANAGEMENT

1.2 OVERVIEW OF QUALITY MANAGEMENT PROCESS

1.3 RURAL DEVELOPMENT TECHNOLOGY OFFICE QUALITY MANAGEMENT POLICY IMPLEMENTS THE FOLLOWING REGULATIONS AND

GUIDELINES:

2 RULES OF ENGAGEMENT FOR QA

3 PROJECT SUPPORT

3.1 PROJECT TEAM

3.2 PROJECT ROLE AND ACTIVITIES

3.3 QA ACTIVITIES DURING PROJECT PHASES

3.3.1 What Quality Assurance Does in the Plan Phase

3.3.2 What Quality Assurance Does in the Design Activity Phase

3.3.3 What Quality Assurance Does in the Develop Phase

3.3.4 What Quality Assurance Does in the Testing Phase

3.3.5 What Quality Assurance Does in the Deploy Phase

3.4 STAGE GATE REVIEWS

4 OPERATIONS AND MAINTENANCE SUPPORT

4.1 O&M ROLES AND ACTIVITIES

5 AD-HOC / EMERGENCY SUPPORT

6 USER ACCEPTANCE CRITERIA

6.1 USER STORIES

6.1.1 AC types and structures

6.1.2 Main challenges and best practices for writing AC

7 TESTING

7.1 FUNCTIONAL TESTING

7.1.1 Unit Testing

7.1.2 Sprint Testing

7.1.3 Smoke Testing

7.1.4 Happy Path Testing

7.1.5 Negative Testing

7.1.6 Boundary Value Analysis Testing

7.1.7 Browser Compatibility Testing and Backwards Compatibility Testing

7.1.8 End to End Testing / System Testing

7.1.9 Accessibility Testing

7.1.10 User Acceptance Testing

7.1.11 Regression Testing

7.2 NON-FUNCTIONAL TESTING

7.2.1 Performance Testing

7.2.2 Stress Testing

7.2.3 Load Testing

7.2.4 Vulnerability Testing

7.2.5 Security Testing

7.3 TEST REPORTING

7.4 TESTING TOOLS

8 CONCLUSION

Page | 3

1 QUALITY MANAGEMENT PROCESS

1.1 Purpose of Quality Management

The purpose of Quality Management is to ensure every time a process is performed, the same information, methods, skills, and controls are used and applied in a consistent manner.

IT Systems are dynamic efforts and as such, change is inevitable. One of the greatest challenges to a system’s success is ensuring changes made to a system are fully reviewed and tested. All original system characteristics, design specifications, documentation, and related data are developed by each system, as required by the Rural Development (RD) Systems Development Life Cycle (SDLC) policy, and thereby establish each system’s design, development, and initial operational baselines. Quality Management, within the context of this guide, ensures changes that are requested and developed by Rural Development Technology Office (RDTO) are reviewed, and tested.

This guide establishes the requirement for following uniform best practices for both managing testing of new applications/systems capabilities and modifications to existing applications/systems. These practices work to ensure requirements and acceptance criteria are well defined and new/modified production code is tested in consistent and precise manner.

Overall Quality Management Objectives are to:

• Ensure that all requirements (user stories) have identified Acceptance Criteria (AC) that are testable.

• Provide controls to ensure all changes to systems are thoroughly tested throughout the lifecycle.

• Ensure that all systems components are Section 508 compliant.

• Ensure that all systems meet performance standands based on requirements.

• Provide testing results that are meaningful and understandable.

• Ensure all changes follow the established governance processes.

• Ensure all systems are safe and secure from security vulnerabilities

This document describes how these objectives will be achieved by outlining the procedures for engaging the Quality Assurance (QA) Staff. To best describe these procedures, this document is divided by major activity as identified below:

• Rules of Engagement for QA

• Project Support

• O&M Support

• Ad-hoc / Emergency Support

• User AC

• Testing

Each of these activities will be outlined in further detail with the appropriate guidelines so that management and all other associated personnel involved in change management have a reference resource. The plan will establish roles, responsibilities and documents that are specific to Rural Development’s configuration environment.

Page | 4

1.2 Overview of Quality Management Process

The Quality Management Process (QMP) helps improve the quality of all deliverables. It allows for a boost in confidence of the quality of an organization’s outputs. By implementing quality management, not only does the delivered product increase confidence, but it also leads to organizational success. The QMP allows organizations to accomplish the following:

• Set quality targets to be met

• Define how those quality targets will be measured

• Take actions needed to measure quality

• Identify quality issues and improvements

• Report on the overall level of quality achieved.

By utilizing the QMP, organizations can achieve the following:

• Perform QA

• Undertake Quality Control

• Initiate Quality Improvement

• Implement Quality Management

A QMP is critical within any organization as it helps ensure that the deliverables produced will meet the requirements of the customers. The QMP will help the organization improve the quality of its deliverables.

A QMP is a set of procedures that are followed to ensure that the deliverables produced by a team are “Fit for Purpose”. The start of QMP involves setting quality targets, which are agreed upon with the customer (AC, Performance Thresholds, etc.). A “QA Process”, and “Quality Control Process” are then undertaken, to measure and report the actual quality of the deliverables. As part of the QMP, any quality issues are identified and resolved quickly.

An Organization should implement a QMP any time that it wants to improve the quality of work performed. Whether the organization is producing deliverables as part of a project or operational team, an effective quality management and QA process will be beneficial. By implementing a QMP, the organization can ensure that the deliverables meet the expectations of your customer.

1.3 Rural Development Technology Office Quality Management Policy implements the following regulations and guidelines:

- ISO 9001, Quality Management Systems - Requirements

- ISO 18091, Quality Management Systems – Guidelines for the application of ISO 9001

- ISO 10006, Quality Management – Guidelines to quality management in projects

- ISO 31000, Risk Management 2 Rules of Engagement for QA

For Quality Management to be successful, certain guidelines must be followed for any QA action to be successful. The following rules of engagement are not meant to be restrictive or to defer any quality support, but are in place to ensure the success of all deliverables:

• When a project is identified, QA must be included at its inception.

Page | 5

• All initiating requests for the QA Team must be originated via sending an email to RDTO_QualityAssurance@usda.gov.

• Requesting Team will provide QA all access, roles, and IDs needed to be successful.

• Requesting Team will provide enough resources to implement and test the quality of deliverables.

• Requesting Team will provide appropriate training to the QA Staff so that the QA Team can quickly, efficiently, and effectly test the system for quality.

• All Requirements that need testing will be identified in User Stories, with complete and accurate

User AC.

• A stable testing environment must be available, including providing access prior to QA testing.

As stated above, the above criteria are in place to maximize the success of your deliverables. If the above criteria are met, the QA team will fully engage via a systematic approach, ensuring effective and efficient support for any Quality needs. If the above criteria are not met, the QA Team cannot guarantee requested timelines will be met.

3 Project Support

Many times, in organizations, QA is considered testing. While testing is a key component of ensuring quality is met, it is not the entire picture. Quality is part of the entire SDLC Project Lifecycle. Simply put, quality is about meeting the customers’ requirements, or more specifically quality is conformance to requirements. The project deliverables need to be fit for use. Quality Management and QA are the totality of features and characteristics of a product or service that bears on its ability to satisfy stated or implied needs. Quality Assurance is the degree to which a set of existing characteristics fulfill requirements. When a product meets customers’ requirements and needs, that is considered a quality product. Conversely, if it is not meeting its stated requirements, the product is said to not meet quality expectations. It is the QA Team’s goal to help ensure that projects conform to all identified requirements, and to reveal when Projects are not meeting this threshold.

To ensure the quality of the product, the QA Team works to determine if the project conforms to all stated customer requirements. Quality Management is a process-based approach whose primary objective is to prevent defects in deliverables beginning at the Plan Stage to avoid rework, which increases project costs, and creates schedule delays.

QA can also be viewed as a proactive process, as it places importance on Planning, Documentation, and Guidelines that will be needed to ensure the Product created is quality. This process begins at the very beginning of Projects both Agile and Waterfall, compared to the usual assumption of just end of Project Testing.

3.1 Project Team

Once a new Project is identified, the QA Team must be immediately notified. This can be accomplished by sending an email to the QA Team, or by scheduling a meeting with the QA Team.

The QA Team will meet with the Project Owner and/or the Project Manager. A QA Resource (and back-

up) will be assigned to the Project for the entire lifecycle. The assigned QA Resource will provide weekly status updates to the Test Architect and will be the QA Point of Contact (POC) for said Project.

If the Project Team, has any issues with the QA Resource assigned, please contact the Test Architect, and the Test Architect will work with all parties to resolve any concerns.

mailto:RDTO_QualityAssurance@usda.gov

Page | 6

3.2 Project Role and Activities

The Assigned QA Resource will support the Project Team in the following ways:

• Will attend all Project Meetings

• Will work with Project Team to ensure Project Requirements have proper User AC

• Will support Development Sprint Testing to ensure delivered User Stories meet the stated User

AC

• Will raise any concerns or risks and will escalate concerns as needed when Project is not meeting expected levels of Quality to the Project Manager, and Produce Owner

• Will train remaining QA Staff when full QA Support is needed for Regression, and End to End

Testing

3.3 QA Activities during Project Phases

Here are examples of activities QA can perform during the various project phases. The below activities provide insight to how QA helps projects become successful in delivering quality results. These activities are for both Agile and Waterfall Projects.

3.3.1 What Quality Assurance Does in the Plan Phase

Gathering requirements is a key step in the SDLC process. There are many advantages of having QA involved in this phase:

• Product knowledge – Product knowledge is essential for QA to do the job to the best of their ability. QA will spend their time going over the user stories, wireframes, and other documents to visualize workflows, understand stakeholder’s goals.

• Identify logical conflicts and discrepancies – As QAs understanding of the project grows with reviewing the project’s documentation, QA may find logical conflicts and discrepancies early in the process.

• Identify poorly defined or unaddressed elements – QA may see “holes” in workflows, permissions, user roles, etc. QA may also see a lack of definition to things like user validations, input validations, text field requirements, etc.

• Provide feedback on user stories – Complicated stories may be difficult to understand, which makes coding and writing test cases more difficult. Based on experience, QA can provide feedback and offer suggestions to simplify user stories.

• Ask questions – As issues come up, QA can start asking questions immediately or point out the issues they see. It’s much easier and more cost effective to get issues addressed before coding begins.

3.3.2 What Quality Assurance Does in the Design Activity Phase The Design Activity Phase is where all the requirements are pieced together to establish the specific workflows, user roles, permissions, etc. and to get a vision of what the final product should look like and how to get there. This process is much easier if QA was involved in the previous phase. Even if you manage to catch every possible issue in the Requirements Phase, which probably won’t happen, the role of QA in the Design Phase is still vital.

• They are a resource – Assuming QA been involved in the Plan Phase, they can become a tremendous resource for the team members who are working out the design of the product. When

Page | 7 the Project Team is unsure how different elements relate to each other, they can reach out to QA prior to see if the intent of the user story is matching the customers requirements.

• Writing the test plan in parallel to the Software Design – This is a great time to write the Test Plan. QA might’ve already identified many points of possible failure. As they see the actual design and the Development plan comes together, QA can design the Test Plan to follow the same flow, so they are always in the same place as the developers. This makes for a more streamlined project.

• Design review – Being able to review the design during this phase allows QA to validate it. QA can help identify any logical conflicts and discrepancies in the design before the developers begin their work.

3.3.3 What Quality Assurance Does in the Develop Phase

During this phase, QA can begin testing. Even during the sprints of an Agile process, there are specific things QA can do to overcome issues and identify defects before they become a major problem.

• Reviewing the User Stories in the current sprint – By reviewing the user stories, QA can confirm that the development is heading the right direction. They can also identify any divergence from the user path.

• Review and writing Test Cases for the current sprint – Reviewing and writing test cases is not only the best way to do the testing itself, it’s also a great way to dissect the requirements on a “microscopic” level. Test Cases are written for each story, and ensure the User Acceptance Criteria is fully tested. The User Acceptance Criteria must map to the requirements.

• Being available and accessible to the Developers – The Developers are going to have questions once they dive into the code. QA can be available and help.

• Testing the software as soon as it is “Done” – When a developer finishes a feature, QA can test it immediately. However, time is of the essence since the developer might have already moved on to the next task.

• Communicating when QA considers the feature as “Done” – Letting the team know when they’ve determined that the feature is working as intended is important. This lets the team fully focus on the next task on their list and lets the project leadership know the true progress of the project.

3.3.4 What Quality Assurance Does in the Testing Phase

This is where everything comes together. Assuming QA has been able to test throughout the process thus far, this step should consist of working on things that may have been missed or not investigated due to time constraints in other phases.

• Edge Case Testing – All of those things that users will probably never do, now is the time to address them. QA can test inputs that will probably never be used, using the application for things it’s not intended to be used for, etc.

• Cross Browser/Cross Operating System Testing – Often, the software may have only been tested using the same browser and operating systems thus far. Additional browser and OS testing will be required during this stage.

• Defect Fixes/ Regression – If the developers haven’t had time to focus on defect fixes, they will now be able to do so. Much of QA’s time will be used to verify defect fixes and doing regression tests.

Page | 8

3.3.5 What Quality Assurance Does in the Deploy Phase

Most of the testing during this phase is focused on User Acceptance Testing (UAT). This is when the client sees the finished product and gets to have a “hands on” experience with the product. The software development team is not only being judged by the quality and functionality of the product; they are also being judged based on the quality of the work they’ve done. A bad experience in this phase could result in the client or stakeholder being dissatisfied about the overall project. What can QA do to help?

• Work with the team to address defects found in UAT quickly – Most clients understand that it is impossible for a provider to produce defect-free software. What the client really looks at is how quickly the defects are fixed. QA stays engaged with the team and works with them to address the defects.

• Are those defects? – QA can be advocates for UAT testers, who may not understand how the software works. Since QA has been involved every step of the way, they can serve as a subject matter expert in this area in support of the client. When a defect is reported, they would validate it with the requirements before the rest of the team is alerted. The defect may just be a misunderstanding with the requirements or something that is out of the project’s scope.

• Exploratory Testing – While UAT is going on, QA can continue to do exploratory testing. This will ensure that the overall project is successful.

3.4 Stage Gate Reviews

The QA Team will have a voting presence on all Stage Gate Reviews (SGR) for all Projects. The Test Architect will provide the Approve/Reject Vote for each SGR based on the quality of the Project. Prior to each SGR, the Test Architect will meet with the Assigned QA Project Team Resource and review all deliverables needed for QA Review, and will provide a Vote that is consistent based on the Quality for the Project. If the Project is meeting the expected Standards, QA will provide an Approval Vote to continue to the next Project Phase. However, if the Project is not meeting the expected Standards, QA may either provide a Reject Vote, or Provide a Provisional Approve Vote if the lacking standards are identified as being resolved within a timely fashion.

Below are the requirements for QA to provide an Approval Vote:

Development Readiness SGR:

• Project Team must meet with QA

• QA will review the Project Plan, Integrated Master Schedule, Release Backlog, and the

Functional and System Requirements documentation, and any comments/changes should be made prior to the Development Readiness SGR.

• If any User Stories are created at this point, all must have User Acceptance Criteria Deploy Readiness SGR:

• All User Stories must be completed

• All Tests have been performed

• All Defects have been corrected/closed, or agreed to be placed in backlog. However, backlog must be approved by the O&M Team.

• No Section 508 defects exist

• All Performance Metrics are met.

Page | 9

• Customers have completed User Acceptance Testing, and provided User Acceptance Signoff

4 Operations and Maintenance Support

Once a Project has been successfully completed, the deliverables from the project enter a new phase in the SDLC, which is ongoing operations, and many times there is the need to provide maintenance on these operations.

Ideally the assigned QA Resource will seamlessly transition from the Project Team, and then continue to provide support to O&M for the system. However, Rural Development was founded in 1994, and even prior to that many systems already existed. Due to this, if QA support is necessary, O&M will need to follow the same initiating actions as Project Support.

The QA Team understands the above and ideally, if funding and staffing do become available, would have at minimum 2~3 staff that will provide only Operations and Maintenance Support. These team members are in addition to the already identified Project QA Staff.

4.1 O&M Roles and Activities

Given current funding and staffing, for O&M, QA Staff will supply the following support:

• QA will have a voting Presence on the Technical Review Board.

If funding and staffing become available, then QA could provide the following additional support:

• QA will identify an O&M QA Resource to support every Change Request that is approved via the TRB.

• The Assigned O&M QA Resource will work with the O&M Team to ensure each approved Change has User AC.

• Once the Change Request has passed Unit Test, it will be assigned for QA Review. The assigned O&M QA Resource will review the Change Request, and either Approve for User Acceptance Testing (when required) or will Return the Change Request for any Defects have found. The criteria for Approval or Rejection is the User AC.

• If the Change Passes QA Review, the O&M QA Resource will create a Test Case/Script for User Acceptance.

5 Ad-hoc / Emergency Support

For times where the QA Team may need to provide ad-hoc or emergency support, please send an email to the QA Team at RDTO_QualityAssurance@usda.gov. If the need is after normal business hours, please contact the RDTO Test Architect. The QA Team will work with the requestor to address and provide the support needed.

6 User Acceptance Criteria

For both DM&E (Projects), and O&M, the term User AC has been referenced, and for those who are new to working with QA personnel, this can be a confusing subject. Acceptance Criteria (AC) are the conditions that a software product must meet to be accepted by a user, a customer, or other system. They are unique for each user story and define the feature behavior from the end-user's perspective. Many times, the organization will just say that a Project, or a change needs to meet the requirements given.

mailto:RDTO_QualityAssurance@usda.gov

Page | 10

Also, many times an organization will just provide a large gathered requirements document and ask that test scripts be created from them. While this can be an acceptable way to conduct business, the Agile Development Strategy and QA has refined this process.

6.1 User Stories

In the IT World, Requirements need to be broken down into digestible pieces of information. In Agile Development, this is accomplished by writing User Stories. A User Story is a tool used in Agile Software Development to capture a description of a software feature from an end-user perspective.

Why use User Stories? Requirements change as teams and customers learn more about the system as the project progresses. It’s not realistic to expect Project Teams to work off a static requirements list, and then deliver functional software days, weeks, and even months later.

A user story is a lightweight method for quickly capturing the "who", "what" and "why" of a product requirement. In simple terms, user stories are stated ideas of requirements that express what users need.

User stories are brief, with each element often containing fewer than 10 or 15 words each. User stories are "to-do" lists that help determine the steps along the project's path. They help ensure that the process, as well as the resulting product, will meet the necessary requirements.

A user story is defined incrementally, in three stages:

• The brief description of the need

• The conversations that happen during backlog grooming and iteration planning to solidify the details

• The tests that confirm the story's satisfactory completion

The acronym INVEST helps to remember a widely accepted set of criteria, or checklist, to assess the quality of a user story. If the story fails to meet one of these criteria, the team may want to reword it, or even consider a rewrite.

A good user story should be:

• Independent: Should be self-contained in a way that allows to be released without depending on one another.

• Negotiable: Only capture the essence of user's need, leaving room for conversation. User story should not be written like contract.

• Valuable: Delivers value to end user.

• Estimatable: User stories must be able to be estimated so it can be properly prioritized and fit into sprints.

• Small: A user story is a small chunk of work that allows it to be completed in about 3 to 4 days.

• Testable: A user story must be confirmed via pre-written AC.

The success of any Project or O&M Activity depends on the ability of a development team to meet the needs of the customer. The communication between the customer and the Project Team or O&M Team plays a vital role in delivering a solution that fits product requirements. Issues arise if customers explain their needs too vaguely and the team can’t understand clear requirements.

For example, imagine that a customer has a request to enable customers to search for a product in an online bookstore by categories. The customer expects to have a clear interface with category links to click on (e.g. fantasy, non-fiction, historic, etc.). After two weeks of development, the development team produces a search bar feature where Customers must type in the category they are interested in, instead of browsing pre-listed categories. While this works, the customers initial goal was to expose all

Page | 11 available categories and let users explore further. This is when high-quality software documentation could help avoid the problem.

As mentioned above, AC are the conditions that a software product must meet to be accepted by a customer. They are unique for each user story and define the feature behavior from the Customers perspective. Well-written AC helps to avoid unexpected results in the end of a development stage and ensure that all stakeholders and customers are satisfied with what they get.

Clarifying the stakeholder’s requirements is a high-level goal. The following are a breakdown of AC purposes:

• Feature scope detail. AC define the boundaries of user stories. They provide precise details on functionality that helps the team to understand whether the story is completed and works as expected.

• Describing negative scenarios. Your AC may require the system to recognize unsafe password inputs and prevent a user from proceeding further. Invalid password format is an example of a so-called negative scenario when a user does invalid inputs or behaves unexpectedly. AC define these scenarios and explain how the system must react on them.

• Setting communication. AC synchronize the visions of the client and the development team.

They ensure that everyone has a common understanding of the requirements: Developers know exactly what kind of behavior the feature must demonstrate, while stakeholders and the client understand what’s expected from the feature.

• Streamlining acceptance testing. AC are the basis of the user story acceptance testing. Each acceptance criteria must be independently testable and thus have a clear pass or fail scenarios.

They can also be used to verify the story via automated tests.

• Feature estimation. AC specify what exactly must be developed by the team. Once the team has precise requirements, they can split user stories into tasks that can be correctly estimated.

6.1.1 AC types and structures

AC can be written in different formats. There are two most common ones, and the third option is to devise your own format.

• Scenario-Oriented (Given/When/Then)

• Rule-Oriented (checklist)

• Custom Formats

As the first and the second formats have very specific structures, so this guide will mostly focus on them. However, you may find that other formats fit your product better, so this guide will briefly touch on them as well.

https://www.altexsoft.com/blog/engineering/striking-a-balance-between-manual-and-automated-testing-when-two-is-better-than-one/

Page | 12

6.1.1.1 Scenario-oriented AC

Scenario-oriented format of writing AC is known as the Given/When/Then (GWT) type.

• Given is the precondition

• When is when some action is expected to be performed

• Then is what the expected result is

This approach was inherited from behavior-driven development (BDD) and provides a consistent structure that helps testers define when to begin and end the testing of a particular feature. It also reduces the time spent on writing test cases as the behavior of the system is described upfront.

Each AC written in this format has the following statements:

1. Scenario – the name for the behavior that will be described

2. Given – the beginning state of the scenario

3. When – specific action that the user makes

4. Then – the outcome of the action in “When”

5. And – used to continue any of three previous statements

When combined, these statements cover all actions that a user takes to complete a task and experience the outcome.

See Examples below:

Example 1

User story: As a user, I want to be able to recover the password to my account, so that I will be able to access my account in case I forgot the password.

Scenario: Forgot password

Given: The user has navigated to the login page

When: The user selected forgot password option

And: Entered a valid email to receive a link for password recovery

Then: The system sent the link to the entered email

Given: The user received the link via the email https://en.wikipedia.org/wiki/Behavior-driven_development

Page | 13

When: The user navigated through the link received in the email

Then: The system enables the user to set a new password

Example 2

User story: As a user, I want to be able to request the cash from my account via an ATM so that I will be able to receive the money from my account quickly and in different places.

AC 1:

Given: that the account is creditworthy

And: the card is valid

And: the dispenser contains cash

When: the customer requests the cash

Then: ensure the account is debited

And: ensure cash is dispensed

And: ensure the card is returned

AC 2:

Given: that the account is overdrawn

And: the card is valid

When: the customer requests the cash

Then: ensure the rejection message is displayed

And: ensure cash isn’t dispensed

6.1.1.2 Rule-oriented AC format

In some cases, it’s difficult to fit AC into the Given/When/Then structure. For instance, GWT would hardly be useful for the following cases:

Page | 14

• You’re working with user stories that describe the system level functionality that needs other methods of QA.

• The target audience for AC doesn’t need precise details of the test scenarios.

• GWT scenarios don’t fit to describing design and user experience constraints of a feature.

Developers may miss several critical details.

These cases can be addressed using the rule-oriented AC format.

The rule-oriented form entails that there is a set of rules that describe the behavior of a system. Based on these rules, the following scenarios can be drawn.

Usually, criteria composed using this form look like a simple bullet list. Let’s have a look at an example.

Example

User story: As a user, I want to use a search field to type a city, name, or street, so that I could find matching hotel options.

Basic search interface AC

• The search field is placed on the top bar

• Search starts once the user clicks “Search”

• The field contains a placeholder with a grey-colored text: “Where are you going?”

• The placeholder disappears once the user starts typing

• Search is performed if a user types in a city, hotel name, street, or all combined

• Search is in English, French, German, and Ukrainian

• The user can’t type more than 200 symbols

• The search doesn’t support special symbols (characters). If the user has typed a special symbol, show the warning message: “Search input cannot contain special symbols.”

6.1.1.3 Other formats

Most user stories can be covered with Rule Orientated, and Scenario Orientated formats. However, Custom AC can be created if they are written in plain English, often even in plain text, serve their purpose, and cannot be misinterpreted.

The Assigned QA Resource member will work with the Project Team to determine the best method of writing AC, and ultimately the goal will be for it to be consistent across the project.

https://www.altexsoft.com/whitepapers/quality-assurance-quality-control-and-testing-the-basics-of-software-quality-management/

Page | 15

6.1.2 Main challenges and best practices for writing AC

While the QA Team is here to help facilitate this information, it is important to have an understanding of the AC, once Project Teams and even for O&M start functioning with AC in mind, the process of being able to produce testable functional code that the Customers can accept becomes the key to functioning more efficiently.

Below are some QA Best Practices:

• Document criteria before development. AC must be documented before the actual development starts. This way, the team will likely capture all customer needs in advance. In the beginning, it’s enough to set the criteria for a small number of user stories to fill approximately two sprints worth of work. They must be agreed by both parties. Then the documented AC are used by developers to plan the technical process.

• Don’t make AC too narrow. AC can be way too specific living little to no maneuver options for developers. To avoid this, remember that AC must convey the intent but not a final solution.

Moreover, narrow AC may be bereft of multiple user behaviors that aren’t covered.

• Keep your criteria achievable. This point closely intersects with the previous one. Effective AC defines the reasonable minimum chunk of functionality that is deliverable. There is a risk of being overwhelmed by small details which would cause the Project Team to become buried with hundreds of small tasks.

• Keep AC measurable and not too broad. Broad AC makes a user story vague. Effective AC must outline the scope of work so that the developers can plan and estimate their effort properly.

• Avoid technical details. As mentioned, AC must be written in plain English. This will make them clear and easy to understand for everyone: Stakeholders or managers may not have technical background.

• Reach consensus. The same problem may can be solved differently by a team and stakeholders, depending on their vantage points. Ensure that the AC has been communicated to the stakeholder and a mutual agreement has been reached. Team members should be included in this mutual agreement.

• Write testable AC. This will allow testers to verify that all requirements were met. Otherwise, developers won’t understand if the user story is completed.

7 Testing

The testing that the QA Team supports is multi-faceted. There is Functional Testing, and Non-functional testing, each of which include numerous types of tests with different functions. These tests are performed as necessary for all Projects and O&M. Each of these will be expanded upon below:

Page | 16

7.1 Functional Testing

Functional Testing is a QA process that bases its test cases on the specifications of the software components under test. Functions are tested by feeding input and examining output. Functional testing usually describes what the system does. Functional Testing can be broken down into the following areas:

7.1.1 Unit Testing

Unit Testing tests an individual software component or module. It is typically done by the programmer and not by testers, as it requires detailed knowledge of the internal program design and code. It is mentioned however, as it typically needs to be completed successfully prior to the Sprint Testing that QA will perform.

7.1.2 Sprint Testing

Sprint Testing is where QA will test each individual User Story that has completed Unit Testing. QA will test each story by ensuring the AC are all met. If there are any AC that are not met, QA will report them and return the User Story back to the Developer.

7.1.3 Smoke Testing

Whenever a new build is provided by the development team, the QA team validates the build and ensures that no major issues exist. The QA team ensures the build is stable, and a detailed level of testing is carried out further. Smoke Testing checks that no showstopper defects exist in the build. This can include some simple tests, like making sure each page loads, and that simple acts like saving data on the page functions properly.

7.1.4 Happy Path Testing

The objective of Happy Path Testing is to test an application successfully on a positive flow. It does not look for negative or error conditions. The focus is only on the valid and positive inputs through which the application generates the expected output.

7.1.5 Negative Testing

Testers having the mindset of “attitude to break” and using Negative Testing will test the system with the intention to break it. A Negative Testing technique is performed using incorrect data, invalid data, or input. It validates if the system throws an error of invalid input and whether the system behaves as expected.

7.1.6 Boundary Value Analysis Testing

Boundary Value Analysis Testing usually goes hand in hand with Negative Testing. This type of testing checks the behavior of the application at the boundary level. Boundary Value Testing checks if defects exist but tests a different range of numbers. There is an upper and lower boundary for each range and testing is performed on these boundary values. If testing requires a test range of numbers from 1~500 then Boundary Value Testing is performed on the values at 0, 1, 2, 499, 500, and 501, etc. This type of testing is excellent for web applications and databases, as it can test the limits of web page fields and help detect if there is a disconnect between values the front-end webpage expects, and what the database will accept as well.

7.1.7 Browser Compatibility Testing and Backwards Compatibility Testing These two types are used to test how web applications function with a combination of different browsers or browser versions. For instance, ensuring the Web Applications works with Firefox, Chrome, Edge, etc.

Page | 17

7.1.8 End to End Testing / System Testing

End to End and System Testing involves testing a complete application environment in a situation that mimics real-world use, such as interacting with a database, network communications, or other hardware, applications, or systems, when appropriate. It also tests all the requirements/user stories/user AC. This may also be known as Integration Testing.

7.1.9 Accessibility Testing

The aim of Accessibility Testing is to determine whether the software or application is accessible for disabled people or not. For government systems, this is typically called Section 508 testing. Various checks are performed such as font size, color, and contrast for color blindness, and using screen readers for those who are blind.

7.1.10 User Acceptance Testing

An Acceptance Test is performed by the customer and verified whether the end to end flow of the system meets the business requirements or not. The customers accept the software only when all the features and functionality work as expected. This is typically the last phase of testing prior to the software going into production. There can be smaller UAT performed by the Customer for individual sprint testing.

7.1.11 Regression Testing

Regression Testing is defined as a type of software testing to confirm that a recent program or code change has not adversely affected existing features. Regression Testing is nothing but a full or partial selection of already executed test cases which are re-executed to ensure existing functionalities work fine. This testing is done to make sure that new code changes should not have side effects on the existing functionalities. It ensures that the old code still works once the latest code changes are done.

7.2 Non-functional Testing

Non-function Testing is the testing of a software application or system for its non-functional requirements. This can include the way a system operates, rather than specific behaviors of the system.

This contrasts with Functional testing as it is not testing against requirements. Non-functional Testing is more in line with reliability and scalability of the system. Non-functional Testing can be broken down into the following areas:

7.2.1 Performance Testing

Performance Testing is often used interchangeably with Stress and Load Testing. Performance Testing is done to check whether the system meets the performance requirements. Different performance and load tools are used to do this testing. An example would be how fast the system performs uploads and downloads of attachments and whether the time it takes to perform such actions meet the acceptable timeframes.

7.2.2 Stress Testing

Stress Testing is done when a system is stressed beyond its specifications to check how and when it fails. This is performed under heavy load like putting large number beyond storage capacity, complex database queries, continuous input to the system or database load.

Page | 18

7.2.3 Load Testing

Load Testing checks how much load or maximum workload a system can handle without any performance degradation. Load Testing helps to find the maximum capacity of the system under specific load and any issues that cause software performance degradation. Load testing is performed using tools like JMeter.

7.2.4 Vulnerability Testing

Typically performed by Cybersecurity, Vulnerability Testing involves identifying weaknesses in the software, hardware, and network. Systems are tested to check for Malicious programs, such as a hacker taking control of the system, scanning to determine if the system is vulnerable to such kind of attacks, viruses, and worms. It is necessary to check if systems are vulnerable prior to approval for Production. If critical defects or flaws are identified, the system will not be approved for deployment until all are remediated.

7.2.5 Security Testing

Typically performed by Cybersecurity, Security Testing is done to check if the software or application is secure from internal and external threats. This testing includes identifying how well the application safeguards against malicious programs, viruses, hacking, or other threats. Testing involves identifying weaknesses and if safeguards are in place to help decrease exposure. Safeguards might include incorporation of appropriate authorization and authentication technologies or other preventive measures.

7.3 Test Reporting

As the above demonstrates, there are a multitude of testing activities that the QA Team can support.

Each of these, where needed, can be tracked via different reports. Most of the functional testing will be provided within the User Stories themselves, and the Non-functional Tests will be provided via a combination of Excel, and Word documentation. QA will provide the Test results to the Project Manager, and if need be, the Project Team may request a follow-on review meeting. If the test reports provided do not meet expectations, please let the QA team know.

7.4 Testing Tools

The Quality Assurance Team will utilize many tools for testing. Some of the tools are listed below, however QA is always working to optimize their ability to support the project teams, and the tools are subject to change.

• JIRA – QA will utilize JIRA to track the User Stories, Acceptance Criteria, Defects found etc.

• Confluence – QA will utilize Confluence to track supported documentation

• Bitbucket – QA will store any Automated Test Scripts within Bitbucket

• Selenium IDE – QA will use Selenium IDE to support the creation of Automated Test Scripts

• Jenkins – QA will use Jenkins to support the use of Automated Test Scripts

• ANDI – QA will use ANDI to test for Section 508 compliance.

• JAWS – QA will use JAWS to support Section 508 Testing

• NVDA – QA will use NVDA to support Section 508 Testing

• Microsoft Office – QA will use Microsoft Office to track emails, write test cases, etc.

• JMeter – QA will use JMeter to support Performance Testing

Page | 19

8 Conclusion

The goal of Quality Management is to support the creation, and release of healthy systems. When QA is involved early, the project teams and customers will see an increase in value. Defects will be identified earlier in the project, requirements will be groomed more quickly, and ultimately the project will be more successful. The QA Team takes this approach seriously. If you need more information, please reach out to the RDTO Test Architect.

3.4 Stage Gate Reviews

6.1.2 Main challenges and best practices for writing AC

8 Conclusion