Attachment J.7_PMO Quality Assurance Plan.doc

Business Relations and

Project Management Office Quality Assurance Plan Version 5.0 August 14, 2009

TABLE OF CONTENTS

11.

Introduction

11.1 Purpose and Structure of this Document

21.2 Scope of Document

21.3 Stakeholders

31.4 Document References

42.

Project Quality Management

42.1 Overview of Quality Management Program

42.2 Role of Quality Assurance in PMO

42.3 Quality Organization

52.4 Quality Roles and Responsibilities

73.

Project QA Description

73.1 QA Tools, Techniques, and Methods

83.2 Standards and Guidance

94.

Quality Activities

94.1

94.2 Document Review

114.3 Peer Review

154.4 Code Review

184.5 Testing Quality Assurance

314.6 Audits

404.7 Process Reviews

425.

Quality Planning and Scheduling of QA Activities

436.

Problem Reporting and Corrective Actions

457.

Quality Measurements

457.1 Quality Measures

468.

Quality Records

489.

Training

49Appendix I – Guidance for Completing the Project-Specific Quality Assurance Plan Addendum

51Appendix II – Document Scoring Guidelines

54Appendix III - Glossary of Quality Terms

55Appendix IV – Acronyms List

TABLE OF TABLES

1Table 1: Sections of Document

3Table 2: Stakeholder Roles

3Table 3: Document References

7Table 5: QA Techniques

8Table 6: Standards and Guidance

12Table 8: Peer Review Information

16Table 10: Code Review Information

18Table 12: Unit Testing Information

20Table 14: System Testing Information

23Table 16: Performance Testing Information

25Table 17: Performance Testing QA Instructions

26Table 18: Section 508 Information

28Table 20: UAT QA Information

32Table 22: Functional Requirements Audit

33Table 23: Preliminary Design Audit

34Table 24: Critical Design Audit

35Table 25: Test Readiness Audit

37Table 26: Release Readiness Audit

39Table 27: End-of-Phase Audits

40Table 28: Process Reviews

41Table 29: Required Process Reviews

45Table 31: PMO Quality Measures

46Table 32: Quality Records

49Table 33: Format for Project-Specific QAP Addendum

51Table 34: Document Scoring Guidelines

54Table35: Glossary of Quality Terms

55Table 36: Acronyms List

TABLE OF FIGURES

4Figure 1: PMO Quality Organization

14Figure 2: Peer Review Process

15Figure 3: Code Development Process

17Figure 4: Code Review Process

19Figure 5: QA Steps in Unit Testing

22Figure 6: QA Steps in System Testing

24Figure 7: QA Steps in Performance Testing

27Figure 8: QA Steps in Section 508 Testing

30Figure 12: QA Steps in UAT

43Figure 13: PMO Quality Action Log

Document Control Page

Current File Location: Document1 Document History Document

Version

Number * Version Description/

Change Description

Version

Used ** Date

Completed

(mm/dd/yyyy) File Location

(Full Path and File name) Signature

Approval

Secured *** (mm/dd/yyyy)

A. Dosen-Black

K. Jastrzemski

08/14/2009

* Be sure to update the document version in the document header with each new version

** Refer to template version below

*** Not required for all templates

1. Introduction

1.1 Purpose and Structure of this Document

The general purpose of this document is to specify policy and provide guidance for quality assurance in the Business Relations and Project Management Office (PMO). The specific objectives of this document are:

· To ensure that project participants fully comply with organizational standards and procedures for quality management

· To improve quality by requiring monitoring of project processes and associated project documents

· To ensure that any issues or findings that may affect quality are identified so they can be addressed Table 1 identifies all of the sections that comprise this document and describes the specific content of that section.

Table 1: Sections of Document

· Describes the technical, documentation, testing, and development standards that apply to PMO projects

· Specifies how these measures will be reported

· Provides standards for the retention of these records

1.2 Scope of Document

This document is the QAP for PMO. All PMO-managed projects are subject to the process detailed in this guide. It applies to all the deliverables, work products, and artifacts associated with those projects, throughout the project life cycle. Please note that although some portions of the document pertain only to software development and implementation projects, such as testing quality assurance, most processes and standards described here can and will be applied to all PMO projects. This includes major internal initiatives as well as work performed for OCC customers.

According to the OCC Systems Development Life Cycle (SDLC) v3.1, QAPs are required for all projects. The SDLC provides a template for these required QAPs. This QAP is based on that template. For smaller and less complex projects, this QAP may serve as the QAP, as long as all quality events are scheduled and included in the project’s Integrated Master Schedule (IMS). For larger and more complex projects, this document, with a brief project-specific addendum, satisfies the requirement for a QAP. The Senior Program Manager and Program Manager, in consultation with the Quality Manager, will determine whether this QAP will suffice, or whether a project-specific addendum is to be prepared. For more information about how to develop the project-specific addendum, please see the guidance in Appendix I.

1.3 Stakeholders

Table 2 identifies the major stakeholders associated with this plan, along with their high-level roles. Many of these stakeholders are directly involved in the execution of quality processes. These specific roles are identified and defined throughout this plan.

Table 2: Stakeholder Roles

1.4 Document References

PMO used several overarching sources to develop this document, including the OCC SDLC v3.1 framework and the Project Management Institute’s Project Management Body of Knowledge (PMBOK). In addition, PMO used the OCC Comptroller’s Memorandum regarding management of information technology systems development projects, dated July 2007. This memorandum provides specific guidance on how quality management responsibilities will be divided between Information Technology Services (ITS) and the OCC PMOs. Table 3 provides specific document references for these sources.

Table 3: Document References

2. Project Quality Management

2.1 Overview of Quality Management Program

The PMO Quality Assurance Program is comprehensive and includes all critical QA activities, including the following:

· Document reviews

· Peer review

· Code reviews

· Testing quality assurance

· Technical audits

· End-of-Phase audits

· Process reviews

PMO supports these direct QA activities with a number of QA infrastructure activities designed to ensure the smooth operation of the quality program. These include the following:

· A quality planning requirement that ensures quality activities are included in the project schedule and appropriate resources are provided for these activities

· A process for reporting, tracking, and closing quality findings

· A quality measurement program

· Quality training for PMO staff

2.2 Role of Quality Assurance in PMO

PMO does not intend to create a large QA infrastructure outside the project teams.

Quality assurance is a shared activity among all PMO staff and support contractors. All project team members are required to participate in and support quality activities by reviewing and scoring documents, attending review sessions, reporting quality findings, and other activities. For more information, see Section 2.3 below.

2.3 Quality Organization

The figure below depicts the PMO organization. For more detail about the roles and responsibilities of PMO and other QA participants, see Table 4.

Figure 1: PMO Quality Organization

2.4 Quality Roles and Responsibilities

Table 4: Roles and Responsibilities

· Makes staff available and provides other resources as needed to support QA

· Reviews high-risk documents as needed

· Participates in other critical QA activities as needed, such as release readiness reviews (go or no go determination)

· Works with customers to resolve issues and gain acceptance of documents

· Works with PMs to determine the need for and scope of peer reviews

· Reviews project-specific QAP addenda

· Reviews and ensures follow-up on all test reports and quality findings

· Produces artifacts and work products that are subject to the quality process

· Promotes compliance with the QA Program

· Verifies that required QA activities are conducted in accordance with PMO policy and reports to the PMO Director

· Monitors the findings associated with quality reviews and audits to identify any emerging quality issues

· Tracks all quality findings to closure

· Works with customers to resolve issues and gain acceptance of documents

· Develops project-specific QAP addenda

· Participates in some QA reviews, audits, and evaluations

· Promotes compliance with the QA program

· Ensures responses to deficiency reports from QA reviews and audits

· Produces artifacts and work products that are subject to the quality process

· Reviews document revisions to verify that document comments have been appropriately addressed

3. Project QA Description

3.1 QA Tools, Techniques, and Methods

PMO will use the following techniques for quality assurance. All of the QA activities specified in this plan employ one or more of these techniques.

Table 5: QA Techniques

· Teams conduct code reviews to find coding errors and other quality issues overlooked in code development

· Technical audits and end-of-phase audits are two examples

3.2 Standards and Guidance

The standards and guidance that will be followed for a typical project include the following technical, documentation, testing, and development standards:

Table 6: Standards and Guidance

Please note that this section describes monitoring of processes to determine whether they adhere to standards. It does not describe reviews of the standards themselves.

4. Quality Activities

4.1 Introduction

This section describes the major QA activities PMO performs. It addresses in detail each of the techniques described in Section 3.1, placing them in context and providing additional detail.

This discussion focuses in greatest detail on activities that are 1) specific to QA, and 2) not covered in detail in other planning documents. For example, Section 4.2, Document Review discusses the document review activities in detail but does not cover document preparation activities like outline preparation. Similarly, Section 4.5, Testing Quality Assurance, focuses more on the role of PMO personnel in monitoring testing activity, while focusing less on testing itself. Testing is a QA activity, but it is covered in detail in required SDLC test planning documents.

Because it is difficult to understand a given QA activity unless it is placed in context, this plan presents some detail about the higher-level processes in which the QA activities are embedded, referred to as “core processes”, in order to provide that context. In the process flow diagrams, core process steps are presented in white, while QA process steps are presented in gray. Please note that in some cases, detailed “drill downs” of QA processes are provided in addition to the core flow. When this occurs, a representation of the core process appears in the top left of the diagram, with the “drill down” process step highlighted in grey.

4.2 Document Review

4.2.1 Introduction - The Document Review Process in Context

PMO project teams execute a rigorous document review process for all written deliverables and major written work products and artifacts. This process has three main phases; management (including the Senior PM and PM), QA (including independent QA reviewers), and customer (including personnel from the customer business unit).

In general, each phase of the document review process follows these steps:

1. In advance of document delivery, reviewer availability is confirmed by the PM and the Document Review Schedule is updated in SharePoint

2. The document is uploaded to SharePoint by the author

3. The author or PM uses SharePoint to alert the reviewers that the document is ready for review

4. The reviewer reviews the document and uploads the commented document to SharePoint, including the appropriate metadata

5. Comments are consolidated by the author

6. The author revises the document and addresses all comments

7. The author publishes the comments to SharePoint

8. The author publishes the revised document to SharePoint in anticipation of the next review.

Please note that the above steps are included in each of the management, QA and customer review cycles. The review process is iterative until such time as the PM and Senior Program Manager feel the document is ready to be finalized and formally delivered.

The document review process is part of the larger document preparation process. Please note that although the preparation of the annotated outline is not part of the QA process itself, it is a required part of the document preparation process. It improves the efficiency of the QA process by allowing reviewers to weigh in on scope and structure before drafting begins. As a result, reviewers can concentrate on reviewing the content of the document rather than scope and structure, reducing re-work.

It is also important to note that representatives of customer organizations (business units) may be included in any of the collaborative processes identified above, at the discretion of the Senior Program Manager and Program Manager. It is important to refrain from making judgments about initial quality during these processes to allow collaboration to occur.

The table below provides basic information about the document review process.

Table 7: Document Review Information

Element Definition

· To provide quality scores for documents (where required)

· Annotated outline

· System Development Life Cycle (SDLC) template, if used

· Comment matrices and quality scores

· Program Manager – perform initial QA review and scoring of document, make sure that all QA comments (regardless of source) are appropriately addressed by the author

· QA Reviewers – review and score draft document

· Senior Program Manager – review and score draft document, ensure that all documents are subjected to the review process,

4.2.2 Detailed Description

A detailed description of the document review process can be found in the To-Be Document Review Processes Analysis document (embedded below). This document provides both a visual depiction of the process in BPMN models and narrative descriptions of each activity.

4.3 Peer Review

4.3.1 Introduction

Peer review is a structured process that allows for examination and criticism of a team’s ideas, approaches, analyses, designs, methods, techniques, and artifacts; always includes individuals with specialized knowledge about the subject matter being reviewed. Peer reviews are conceptual reviews conducted to define the purpose of a document or artifact, determine the content and organization of a document, discuss the approach, and/or set expectations. They are generally conducted as a facilitated meeting.

When the item under review is a document or artifact, the peer review should be conducted early in the process of developing the artifact, before any other formal evaluation activities are conducted. It may be held before an outline or draft has been prepared. Alternatively, it may be held to develop an outline or to review a preliminary or “rough draft” of an outline or document.

For system implementation projects, a peer review should be conducted for at least the following written documents:

· System Concept of Operations (ConOps)

· System Development Plan (SDP)

· Security Plan

· Functional Requirements Document (FRD)

· Test Plan

· System Design Document (SDD) or Design Description Document (DDD)

In addition, the PM, in conjunction with the relevant Senior Program Manager, may decide to conduct a peer review for any artifact that meets one or more of the following characteristics:

· Major documents, analyses, concepts, methods, or approaches

· Artifacts that include subject matter in high-risk areas

· Artifacts that include complex subject matter requiring expert review

· Artifacts including subject matter of a sensitive nature

For written documents, the peer review process is part of the larger document preparation process. Figure 4 depicts the document preparation process and shows the placement of the peer review process within it.

To the extent possible, the PM, in consultation with the Senior Program Manager, is to decide during the planning phase precisely which items will be subject to peer review. This will allow time to be built into the project plan for these reviews. However, the PM may decide at any time to conduct a peer review.

Table 8 provides basic information about the peer review process.

Table 8: Peer Review Information

Element Definition

· During the initial concept-development stage for ideas, approaches, analyses, designs, methods, techniques, and artifacts

· Agenda

· Checklist (optional)

· Senior Program Manager – consult with the Program Manager on what requires peer review

· Team Members – participate in review

· Peer Reviewers – participate and provide comments within their areas of professional expertise

· Owner

– address peer reviewer comments

4.3.2 Detailed Description

The figure below shows the peer review process in detail. The process depicted in Figure 2 can be applied to a full range of subjects, including ideas, approaches, processes, analyses, designs, methods, techniques, deliverables or artifacts. The process flow is followed by Table 9 with detailed information about each process step in the figure.

Figure 2: Peer Review Process

Table 9: Peer Review QA Instructions

· The Program Manager or Senior Program Manager is to determine the scope of the review, identify the reviewers, and prepare the agenda.

· A checklist may also be prepared to structure the discussion.

· The parties listed above are to hold the peer review session as a facilitated meeting.

· The “owner” or other participant is to document the session by recording minutes and action items and distributing them to the participants for validation.

· The “owner” is the individual primarily responsible for the idea, approach, process, analysis, design, method, technique, deliverable or artifact under review. For written documents, the owner would be the primary author of the document.

· The owner is to execute the action items associated with the peer review.

· The owner may delegate the action items if appropriate, but bears overall responsibility for their execution.

· The Program Manager or Senior Program Manager is to follow up to make sure action items are executed.

· The Program Manager is to document, monitor, and close any major findings resulting from the peer review that are not resolved at the time of the review.

4.4 Code Review

4.4.1 Introduction

Code review is a technical review that focuses on a system's source code. Teams conduct code reviews to find coding errors and other quality issues overlooked in code development. The code review process is part of the larger code development process. Figure 3 depicts the code development process and shows the placement of the code review process within it.

Figure 3: Code Development Process

As shown in the figure, Code Review is conducted concurrently with code development. It can occur one time or multiple times during development, depending on the complexity of the code, the experience level of the developers, or the extent of findings from previous reviews.

PMO will conduct code reviews for all projects. Code review sessions are facilitated meetings in which members of the project team review code line-by-line. Sometimes participants review all code. However, good results are often achieved by reviewing as little as twenty-five percent of the code. Any code that is not subject to a formal code review is to be subject to peer code review. Peer code review is less formal and occurs when members of the development team review each other’s code on an ongoing basis during the development process.

Table 10 provides basic information about formal code review sessions.

Table 10: Code Review Information

Element Definition

· Development standards

· Code review checklist

· Program Manager – participate in review, ensure required code reviews occur, track and close major findings that cannot be resolved as part of the review process

· Senior Program Manager – participate in review

4.4.2 Detailed Information

Figure 4 shows a “drill down” of the code review process itself, when the code review is performed as a formal, facilitated session. The process flow is followed by Table 11, which provides detailed information about each process step in the figure.

Figure 4: Code Review Process

Table 11: Code Review Instructions

· The Development Lead is to decide what code will be subject to a formal code review; some code may be subject only to peer code review.

· The Development Lead is to prepare a code review checklist to be used in the facilitated session.

· The checklist shall validate that coding standards are met; this includes both naming and coding conventions and efficiency considerations like reusability of code.

· The Development Lead is to conduct the code review facilitated session and document all findings.

· The Program Manager will document, monitor, and close any major findings resulting from the code review that are not resolved at the time of the review.

4.5 Testing Quality Assurance

For software implementation projects, the Program Manager will be actively involved in the testing process. Below, testing quality assurance activities are discussed for each major testing phase.

4.5.1 Unit Testing

During unit testing, the developers on the project team verify that each individual component (created or purchased) performs as specified in the design documents. Throughout the development and testing phases, developers will conduct unit tests. Each developer is responsible for identifying the test data that will be used to execute and confirm the results of the unit tests. Prior to the migration of the product from the development environment, the developer will be required to validate that the unit tests were successfully conducted and the proper results were achieved.

Table 12 provides basic information about the unit testing process.

Table 12: Unit Testing Information

Element Definition

· Development Team – plan test and draft test plan,

· Development Lead – review test plan, confirm test readiness, review test documentation, verify that end-of-cycle criteria are met

· Program Manager – ensure all planned unit testing occurs, track any major findings not resolved during testing cycle

· Units have been successfully evaluated with desired results achieved

Quality assurance of unit testing is part of the larger unit testing process. Figure 5depicts the unit testing process and shows the placement of the QA steps within it. It is followed by Table 13, which provides detail about each QA process step.

Figure 5: QA Steps in Unit Testing

Table 13: Unit Testing QA Instructions

The Development Lead evaluates the Test Plan relative to the following criteria:

· Does the test plan validate the functionality of the unit as designed in specification?

· Has the test plan documented the unit testing environment?

· Does the test plan require isolation for testing of each unit?

· Does each unit test case include an objective, input, and expected outcome?

· Are unit testing deliverables defined?

Confirm Test Readiness (Development Lead)

· The Development Lead is to confirm that all unit testing plans, specifications and environments are in place prior to execution.

· The Development Lead is to verify that unit test outcomes are recorded, including discrepancies.

· The Development Lead is to verify re-testing of activities to address discrepancies.

· The Development Lead is to confirm test records have been checked against specified test completion criteria.

· The Program Manager will document, monitor, and close any major findings resulting from unit testing that are not resolved during the testing process.

4.5.2 System Testing

System testing ensures that system components, including application software, data conversion software, job schedules, system modifications and operations procedures, are technically compatible. System testing validates data flows from point to point and ensures that the components of the application operate together as a whole. It also validates that before the application effectively integrates with other existing or new applications. A typical check would be that record or transaction counts from a sending system are the same as those in the receiving system or that a user can perform the steps required to navigate through the application.

Table 14 provides basic information about the system testing process.

Table 14: System Testing Information

Definition

· Test cases and scripts

· Functional Requirements Document

· System Requirements Specification

· System Design Document

· Verification that end-of-phase criteria are met

· Test Lead –review test plan, prepare for test cycle, confirm test readiness, deploy code, review test documentation, manage defects, verify end-of-phase criteria are met

· Configuration Manager – work with Test Lead to deploy code

· Testers – execute test, document test, report test results, log defects, complete test report

· Technical/Development Team – provide testing technical support, fix defects

· Program Manager– review test results and test report, make sure that all required system testing is conducted, track any major findings not resolved during testing

Quality assurance of system testing is part of the larger system testing process. Figure 6 depicts the system testing process and shows the placement of the QA steps within it. It is followed by Table 15, which provides detail about each process step.

Figure 6: QA Steps in System Testing

Table 15: System Testing QA Instructions

· The Test Lead is to verify that the system test plan is in place.

· The Test Lead is to verify that the system test plan identifies the most important/critical requirements and establishes the correct depth of testing for each functional area.

Confirm Test Readiness (Test Lead)

· The Test Lead confirms that all system test plans and specifications are in place prior to execution.

· The Test Lead is to review system test logs and defect reports.

· The Test Lead is to verify re-testing of system test defects.

· The Test Lead confirms that test records have been checked against specified test completion criteria.

· The Program Manager will document, monitor, and close any major findings resulting from system testing that are not resolved during the testing process.

4.5.3 Performance Testing

Performance testing measures the system’s capability to meet performance requirements under expected normal and peak production conditions and user load levels. Performance testing typically occurs after the system has successfully passed system testing. System performance benchmarks must be established before performance testing can occur.

Table 16 provides basic information about the performance testing process.

Table 16: Performance Testing Information

Definition

· Test Lead – review test plan, prepare for test cycle, deploy code, review test documentation, manage defects, verify end-of-phase criteria are met

· Configuration Manager – work with Test Lead to deploy code

· Testers –execute test, document test, log defects, report test results

· Technical/Development Team – provide testing technical support, fix defects

· Program Manager – review test results and test report, ensure that performance testing is conducted, track any major findings not resolved during testing

Quality assurance of performance testing is part of the larger performance testing process. Figure 7 depicts the performance testing process and shows the placement of the QA steps within it. It is followed by Table 17, which provides detail about each process step.

Figure 7: QA Steps in Performance Testing

Table 17: Performance Testing QA Instructions

· The Test Lead is to verify that performance characteristics are identified (e.g., Response Time, Throughput).

· The Test Lead is to verify the amount of test data needed for each parameter.

· The Test Lead is to verify critical system components.

· The Test Lead is to verify the physical environment (machine configuration, hardware).

· The Test Lead is to verify the performance test scripts (the steps/activities for specific user scenarios).

· The Test Lead is to verify the performance metrics (e.g., Request execution time must not exceed eight seconds).

· The Test Lead is to verify the test cases (e.g., Take a single instance of a test script and gradually add more instances and/or more scripts over time, increasing system load).

Confirm Test Readiness (Test Lead)

· The Test Lead is to confirm all performance testing plans and specifications are in place prior to execution.

· The Test Lead is to verify that results have been captured, compared against each metric’s acceptable or expected level, and documented.

· The test lead is to verify that failed tests are diagnosed and re-tested.

· The Test Lead is to confirm that test records have been checked against specified cycle completion criteria.

· The Program Manager will document, monitor, and close any major findings resulting from performance testing that are not resolved during the testing process.

4.5.4 Section 508 Testing

Section 508 of the Rehabilitation Act of 1973, as amended in 1998, requires that the Federal government provide access to, and use of, information and data to Federal employees and members of the public with disabilities that is comparable to that provided to those without disabilities.

The U.S. Access Board developed functional criteria and technical standards for electronic and information technologies to facilitate implementation of this law. The criteria and standards were adopted into Federal Acquisitions Regulations (FAR) and are located at CFR1194.21-26, 31, and 41. For the purposes of this document, this section will focus only on CFR 1194.21, which outlines the technical standards for software applications and operating systems.

Table 18 provides basic information about the Section 508 testing process.

Table 18: Section 508 Information

Definition

· Test Plan

· Functional and system requirements

· Software Application

· SDD or DDD

· Section 508 Test Team – execute test cases, manage defects, complete test report

· Development Team – provide testing technical support, fix defects

· Configuration Manager – work with project Test Lead to deploy code

· Project Test Lead – review test plan, work with Configuration Manager to deploy code, review test documentation

· Program Manager – verify that required testing has occurred, track any major findings that could not be resolved during the testing process

Quality assurance of Section 508 is part of the larger Section 508 testing process. Figure 8 depicts the Section 508 testing process and shows the placement of the QA steps within it.

Figure 8: QA Steps in Section 508 Testing

Table 19 details the core QA steps for Section 508 Testing.

Table 19: Section 508 Testing Instructions

· The Project Test Lead is to review Section 508 guidelines.

· The Project Test Lead is to verify that the Section 508 testing strategy (with CFR 1194.21 checklist) is in place.

· The Project Test Lead is to verify appropriate end users (represent the experience for users with disability).

· The Project Test Lead is to verify that appropriate tools are in place (manual/automated testing).

Confirm Test Readiness (Section 508 Test Lead)

· The Section 508 Test Lead is to confirm that Section 508 tests are ready for execution.

· The project Test Lead is to review Section 508 test logs and defect reports.

· The project Test Lead is to verify re-testing of Section 508 defects.

· The Section 508 Test Lead is to verify that end-of-phase check is complete.

· The Section 508 Test Lead is to confirm test records have been checked against specified test completion criteria (checklist).

· The Section 508 Test Lead may consider an independent third-party audit for compliance.

· The Program Manager will document, monitor, and close any major findings resulting from Section 508 testing that are not resolved during the testing process.

4.5.5 User Acceptance Testing

The focus of User Acceptance Testing (UAT) is the final verification of required system flow and business functions. UAT provides validation of the end users’ workflow, using the system just as they would in production. It also allows users to provide feedback on system functionality, workflow and performance.

UAT testing not only allows for a final verification of the system’s conformance to requirements and design, but also promotes end-user support and buy-in for the new system by soliciting user input prior to deployment.

Table 20 provides basic information about the UAT process.

Table 20: UAT QA Information

Element Definition

· To verify and provide feedback on system functionality, workflow and performance

· Test cases and scripts

· Functional Requirements Document

· System Requirements Specification

· System Design Document

· Test Analysis Reports

· Change Requests associated with any defect fixes

· Results of re-testing activity performed to validate defect fixes

· Test Lead –prepare for test cycle, deploy code, review test documentation, manage defects, verify end-of-phase criteria are met

· Configuration Manager – work with Test Lead to deploy code

· Testers – execute test, document test, log defects, report test results

· Technical/Development Team – provide testing technical support, fix defects

· Program Manager – review test results and test report, ensure that UAT is conducted, track any major findings not resolved during testing

Quality assurance of UAT is part of the larger UAT testing process. Figure 9 depicts the UAT testing process and shows the placement of the QA steps within it. It is followed by Table 21, which provides detail about each process step.

Figure 9: QA Steps in UAT

Table 21: UAT QA Instructions

· The Test Lead is to verify that the UAT plan is in place, is sound, and contains all necessary information.

· The Test Lead is to verify that the testing environment is adequately described.

· The Test Lead is to make sure that a diverse group of end-users has been retained to execute the test and that they will be adequately trained in testing procedures.

· The Test Lead is to verify that the UAT plan identifies the most important/critical requirements and establishes the depth of testing for each functional area.

Confirm Test Readiness (Test Lead)

· The Test Lead is to confirm that all system test plans and specifications are in place prior to execution

· The Test Lead may confer with the customer to assess test readiness

· The Test Lead is to review UAT test logs and defect reports.

· The Test Lead is to verify re-testing of system test defects.

· The Test Lead is to verify that end-of-phase check is complete.

· The Test Lead is to confirm that test records have been checked against specified test completion criteria.

· The Program Manager will document, monitor, and close any major findings resulting from UAT that are not resolved during the testing process.

4.6 Audits

4.6.1 Technical Audits

PMO will conduct a series of technical audits for each systems implementation project. These audits will be conducted to verify that major system deliverables or artifacts meet or exceed requirements. Below, summary information about each technical audit is presented.

Teams generally conduct these audits as a facilitated session. Also, these audits generally focus on a checklist that key stakeholders subsequently sign to signify concurrence with audit results. For each required technical audit, OCC’s SDLC provides a checklist that PMO may use.

The PM is to determine who will conduct or facilitate each audit session. The PM may choose to facilitate the meeting personally, or may select someone else from project leadership. The PM may select an independent technical expert to facilitate a review.

The following tables provide information on how to conduct and document these audits.

4.6.1.1 Functional Requirements Audit

Table 22: Functional Requirements Audit

Definition

· Note: This audit can be combined with the end-of-phase review for the requirements phase.

· Requirements Traceability Matrix (RTM)

· System Workload Analysis

· Interface Control Document

· Data Management Plan

· System ConOps

· Business Unit Functional Lead – review and concur with review

· Other technical experts – attend and participate in review

· Review checklist completed and signed

· Schedule meeting or meetings

· Conduct meetings

· Resolve or address any findings or areas of nonconformance

· Document meeting and prepare checklist

· Obtain approval signatures

4.6.1.2 Preliminary Design Audit

Table 23: Preliminary Design Audit

Element Definition

· SDD or DDD

· Data Conversion Plan

· FRD

· RTM

· Interface Control Document

· Data Management Plan

· ITS Project Manager –participate in review

· Development Lead – participate in review

· Business Unit representative – participate in review

· Other technical…