Attachment 6 Acceptance and Defect Rubric.pdf

5/27/2020 Version 1.2

Defect Management Guide – Acceptance & Defect Rubric

Solution Delivery Division

Technology Office Rural Development Washington, D.C.

T A B L E O F C O N T E N T S

USER ACCEPTANCE TESTING

Purpose Acceptance Criteria Definition Determination of Acceptance or Non-Acceptance

SOFTWARE DEFECTS

Definition Prioritization Roles and Responsibilities Defect Management and Remediation Process

User Acceptance Testing

User Acceptance Testing (UAT) as defined by numerous standards boards is “formal testing with respect to user needs, requirements, and business processes conducted to determine whether a system satisfies the acceptance criteria and to enable the user, customers or other authorized entity to determine whether to accept the system.”

Purpose

To provide confidence to the customer that the developed product meets both the functional and non-functional requirements, the customer conducts an acceptance test using an acceptance test suite. Once testing is completed, and system acceptance is achieved, the development team expects the customer to sign-off on the product as satisfying the defined requirements.

The test team running an acceptance test suite uses predefined acceptance test procedures or test scripts. These direct the test team which data to use, the step-by-step processes to follow and the expected results. The test team records the actual results and comparison with the expected results as specified in the test script and associated Feature / User Story. If actual results match expected results, the Feature / Use Story is marked “Pass” and is accepted.

Prior to execution of UAT, all parties agree what constitutes system acceptance. Generally, this is the percentage of test cases within the acceptance test suite. If the test case pass percentage meets the system acceptance threshold, then system is accepted.

Acceptance Criteria Definition

Acceptance Criteria are a set of statements, each with a clear pass/fail result, that specify both functional (Intake loan application with up to 5 borrowers) and non-functional (e.g., Borrower Name cannot contain the following special characters ‘”;;./? and be no more than 50 characters in length). These requirements represent “conditions of satisfaction.” There is no partial acceptance: either a criterion is met, or it is not met.

The criteria define the boundaries and parameters of a Feature / User Story. They determine when a story is completed and working as expected. They add certainty to what the development team is building.

Each acceptance criterion must be expressed clearly, in simple language the customer would use, just like the User Story, without ambiguity as to what the expected outcome is: what is acceptable and what is not acceptable. An acceptance criterion must be testable: easily translated into one or few manual/automated test procedures.

Determination of Acceptance or Non-Acceptance

The Government is the sole determiner of pass / fail for each acceptance criterion. The Government has the sole decision if a Feature / User Story passes the acceptance criteria. The Government can at its sole discretion determine that a Feature / User Story is acceptable with a defect or defects. The Government can also at its sole discretion determine that a Feature / User Story can be conditionally accepted with a defect or defects, which are to be remediated in the future.

Software Defects

If the Government determines that a Feature / User Story is acceptable with a defect or defects, then those defects will be added to the defect backlog and prioritized

Definition

A software defect is an error, flaw or fault in a computer program or system that causes it to produce an incorrect or unexpected result, or to behave in unintended ways.

Prioritization

The Government is the sole determiner of defect priority. The defect rubric included below, is an objective description that both the Government and the Contractor can use to determine a defect’s priority. The Defect Triage Team will work to evaluate and determine a defect’s priority. Should there be disagreement between the Government and the Contractor, the Government, at its sole discretion, can determine a defect’s priority. If the Government exercises this power, the Government will provide a written justification explaining the Government’s defect priority decision. The Government’s Contracting Officer will review the Government’s written justification and any written response provided by the Contractor to make final determination.

Roles and Responsibilities

Role Definition Responsibility Product Owner /Requiring Office

Representative from the Department/office that requires the product/program and provides the funding to acquire such.

Provide insight and perspective as to the severity of a defect. Insight and perspective are based on knowledge of the business’s and external customer’s needs. Determine if a proposed work around is acceptable to the Government.

Role Definition Responsibility Defect Triage Team (If/When Required)

Cross functional team that supports and steers the project. Members of this team include COR, Product Manager, Product Owner, Vendor Program Manager, and TO Project Manager.

Additional Representation may include Security, Quality Assurance, or other authorized delegates as needed.

Delegation of authority is authorized.

For those Features / User Stories that fail to meet acceptance criteria, this team has the following responsibilities:

Determine and assign Severity rating

Determine priority of defect within the severity rating category

Technology Office (TO) Management

Includes CIO, DCIO, Solution Delivery Director, Chief Information Security Officer (CISO), and other members of RDTO Sr.

Management Team.

Ultimate authority for all Approvals or Disapprovals of development activities

Ensures compliance with all TO standards, USDA standards, contractual obligations, and Federal Information Technology Laws and Regulations.

Product Manager (Product

Development Manager)

Owner of the assigned product development activities.

Chairs the Defect Triage team (when required)

Calls Defect Triage meetings Ensures all relevant information about a defect is made available to the Defect Triage team.

Responsible for all product and project development activities.

Ensures User Acceptance Testing in proceeding as planned.

Ensures that all decisions are documented.

Tracks status about each defect and its place in the remediation process

Role Definition Responsibility Enforces RDTO SDLC as outlined in the project’s tailoring plan.

Provides written justification for the Government’s determination of a defect prioritization decision when the Defect Triage Team cannot come to agreement.

Presents and reviews the written justification to the CO for resolution with the Contractor’s contracts manager.

Contracting Officer Representative (COR)

An individual, including a contracting officer’s technical representative (COTR), designated and authorized in writing by the contracting officer to perform specific technical or administrative functions.

FAR Part 2.101

Provides written justification for the Government’s determination of a defect prioritization decision when the Defect Triage Team cannot come to agreement. Presents and reviews the written justification to the CO for resolution with the Contractor’s contracts manager.

Contracting Officer

(CO)

Only person who can enter the Government into a binding contractual agreement

Provides final decision for those defects where the Defect Triage Team cannot come to agreement.

Enforces contract terms and conditions

Vendor Product Development

Manager (VPDM)

Vendor’s manager responsible for leading their efforts to develop and deliver the product. Represents the vendor day-to-day and determines other vendor personnel required to attend and participate in the Defect Triage team meeting.

Provides technical assessment of defect

Provides best time estimate to repair defect

Recommends severity rating Reports status of defect repair by severity for defects on the backlog

Defect Management and Remediation Process

The Government requires the contractor to use the current USDA’s RDTO defect management process and tools for development projects.

Severity

Definition Workaround

Production Deployment

Allowed Example Defect QA Environment:

Suggested Actions UAT Environment: Required Actions Level Name

1 Critical / Blocker

A critical defect:

1) Blocks a business function from proceeding, is completely broken, results in a complete shut-down of a functional process, nothing can proceed, and no viable workaround exists, OR

2) Is rated as a Critical cybersecurity vulnerability

No No Launching ‘New Application’ renders error, user cannot proceed.

System will not allow 5 borrowers to be entered. This will prevent the agency from calling credit and will impact the ability to determine if the application should be approved.

Financial calculations are incorrect. One calculation that is important in GUS is the PITI (principal, interest, taxes, and insurance) calculation is incorrect. If the system does not save the tax amount, it would prevent the Agency from determining if the application should be approved or rejected.

Application data (Financial and/or non-financial) is incorrect in the system and affects underwriting.

Address prior to sprint closure by one of the following methods:

Repair the defect in the current Sprint and deliver the Feature / User Story

Declare the Feature / User Story as not complete and repair the defect in next or subsequent Sprint but prior to completing the Product Release.

Address immediately

1) Triage to determine cause of issue

2) Provide estimate to complete remediation

3) Provide periodic progress updates balancing progress report periodicity with need to limit interruption and disruption of defect repair efforts

4) Apply Hot Fix to UAT environment

2 High A high severity defect:

1) Blocks a business function from proceeding and a viable but not acceptable workaround exists. The workaround enables information from the impacted business process to be made

Yes – Only viable during testing to allow testing to progress. Must be fixed before deployment to production.

No User cannot navigate to next page of application but can save and reopen.

“Other Asset” can’t be added due to code issue.

Address prior to sprint closure by one of the following methods:

Repair the defect in the current Sprint and deliver the Feature / User Story

Declare the Feature / User Story as not complete and repair the defect in next or subsequent Sprint but prior to

Balance with current sprint priorities and address prior to release for UAT or to production.

*May be applied as Hot Fix to UAT

Definition Workaround

Production Deployment

Allowed Example Defect QA Environment:

Suggested Actions UAT Environment: Required Actions Level Name available so other business functions can proceed. The workaround is not acceptable to RD other than to allow testing to proceed, OR

2) Is rated as High cybersecurity vulnerability completing the Product Release.

3 Medium A medium severity defect:

1) Causes undesirable behavior in a portion of a business function, the system is still functional, and a viable and acceptable workaround exists. The workaround must be acceptable to Rural Development, OR

2) Is rated as medium cybersecurity vulnerability.

Yes Yes Validation is missing.

Missing edit in user interface. It is caught later in the system. Edit on borrower page is not triggered but caught when application is validated.

If all Critical & High defects have been addressed, then a medium defect should be corrected within current sprint. If remediation is not possible in the current sprint, then the defect should be fixed in subsequent sprints. If something prevents a medium defect from being addressed in the product release, then the defect’s remediation must be addressed as part of a release defect resolution plan.

Add to backlog and work in priority order with other user stories and defects as time allows in current release or a future release.

4 Low A low severity defect:

1) Is a minor flaw that does not impede or impact any business

A workaround might exist but is not required

Yes Cosmetic in nature Field label contains extra space.

Label on the user interface has a typographical error or is misspelled.

If all Critical, High & Medium defects have been addressed, then a low defect should be corrected within current sprint. If

Add to backlog and work in priority order with other user stories and defects as time allows in current release or some future release.

Definition Workaround

Production Deployment

Allowed Example Defect QA Environment:

Suggested Actions UAT Environment: Required Actions Level Name function in the system, OR

2) Is rated as low cybersecurity vulnerability

Error message should be updated. remediation is not possible in the current sprint, then the defect should be fixed in subsequent sprints. If something prevents a low defect from being addressed in the product release, then the defect’s remediation must be addressed as part of a release defect resolution plan.

Terms and Definitions

Term Definition

Test Plan Defines the scope, objectives, approach, QA activities for the project, roles & responsibilities, entry & exit criteria among other things required to test a software application. It is derived from the functional and non-functional requirements.

Test Strategy Outlines the testing approach and is derived from the High-Level Business Requirements. It can include customer communication strategy and engagement.

Test Case Provides step by step instructions for a person to test a software application. It is a written document that used authorized test case form templates.

Test Script Provides a set of instructions to automatically test a software application. It is the automation of a test case.

Test Scenario Defines a process for all possible ways to test a software application.

Test Condition Defines a static rule or rules, i.e., instruction to the tester, to follow for testing a software application.

Testing Procedure Is a combination of test cases logically linked together, for example End-to-End testing or integration-testing.

Test Suite Is a group of test cases used for regression testing and order of test case execution may or may not matter.

Smoke Test Is used by development teams to assess if the main functions are working properly, to reveal simple failures severe enough to prevent release in the next environment (QA, Cert, UAT, Production). Smoke tests are a subset of test cases.

Input Specifications Are the specific values that a test team member executing a test script must enter at a specific step in the test case.

Execution Conditions Defines the conditions in which a particular test item must be executed.

Expected Results The specific results that a tester will see at the end of the test case.