Attachment J.2 SOO.doc

Comptroller of the Currency Office of Management

Business Relations and

Project Management Office (PMO)

ECMP

Statement of Objectives August 31, 2011 Document Control Project Name Economics Computing Modernization Project

Project Acronym

ECMP

Document Title ECMP Statement of Objectives (SOO) Document Date August 31, 2011 Client Organization Economics (ECON)

Table of Contents Introduction

61.1 Project Background

61.1.1 Treasury Department and Office of the Comptroller of the Currency

61.1.2 OCC Economics

71.1.3 The Economics Computing Modernization Project

81.2 Document Purpose

Solution Objectives

92.1 Introduction

92.2 Implementation Objectives

92.2.1 Phases and Stages of Implementation

142.2.2 Success Criteria for the Implementation Phases

152.3 Infrastructure Objectives

162.4 Support-Services Objectives

172.5 Solution SLA Objectives

202.6 Solution Testing Objectives

202.6.1 Pilot Tests

202.6.2 Phase-I Tests

212.6.3 Phase-II Tests

222.6.4 Phase-III Tests

Deliverables and Deliverable Schedule

243.1 Submission of Deliverables

243.2 Deliverable Types

243.3 The PMO Document Review Process

253.3.1 Review Times

263.3.2 Number of Reviewers

273.3.3 Review Guidelines and Scores

273.3.4 Version Requirements for Documents Not Being Scored

273.4 Importance of the Management Review Version

283.5 Remediation of Unacceptable Deliverables

283.6 Required Deliverables

423.7 Deliverable Milestone Dates

423.8 Billing Schedule for Fixed Price Tasks

Performance Measures and Acceptance Standards for Deliverables

454.1 Milestone Timeliness

464.2 Deliverable Quality

474.3 Performance Incentives and Penalties for Deliverables

484.4 Performance Incentives and Penalties Associated with the SLA

Standards and Guidelines

505.1 Solution Use Case Model

505.2 Technical and Performance Requirements Document

505.3 Solution Testing Requirements

505.3.1 Test Environment

505.3.2 Test Tools

515.3.3 Test Execution

515.3.4 Test Reporting

525.4 Status Reporting

525.5 Privacy

525.6 Document Formatting and Style Guide

525.7 PMO Quality Assurance Plan

535.8 Service Provider Information Security Self-Assessment

535.9 OCC Project Resources

Key Personnel

A-1Appendix A Acronyms

List of Figures 9Figure 2–1 Overview of ECM Proposed Implementation Schedule

List of Tables 11Table 2–1 Definition of the Scope of the ECMP Pilot

Table 2–2 ECMP Phase Success Criteria Table 2–3 ECMP SLA Objectives Table 3‑1 Minimum Times for Deliverable Review and Revision (Phases I – III) Table 3–2 Deliverables for the Pilot Table 3–3 Deliverables for Phase I Table 3–4 Deliverables for Phase II Table 3–5 Deliverables for Phase III Table 3–6 Billing Schedule for Pilot Fixed-Price Deliverables Table 3–7 Billing Schedule for Phase-1 Fixed-Price Deliverables Table 3–8 Billing Schedule for Phase-2 Fixed-Price Deliverables Table 3–9 Billing Schedule for Phase-3 Fixed-Price Deliverables Table 4–1 ECMP Performance Objectives for Deliverables Table 4–2 Milestone Timeliness Performance Standards Table 4–3 Deliverable Quality Scores Table 4–4 Deliverable Quality Performance Standards Table 4–5 Contract Performance Incentives and Penalties Table 5–1 Anticipated OCC ECMP Resources Table A–1 Acronyms A-1

1 Introduction

1.1 Project Background

1.1.1 Treasury Department and Office of the Comptroller of the Currency The Department of the Treasury is the primary Federal agency responsible for the economic and financial prosperity of the United States. The Treasury is organized into departmental offices and operating bureaus. The bureaus carry out specifically assigned mission operations. A primary mission operation is the corporate governance of financial institutions.

As a bureau, the Office of the Comptroller of the Currency (OCC) charters, regulates, and supervises national banks to ensure a safe, sound, and competitive banking system that supports both the citizens and the economy of the United States. The Comptroller is a presidential appointee with offices headquartered in Washington, D.C. The Comptroller also serves as a director of the Federal Deposit Insurance Corporation (FDIC).

The OCC is divided into seven functional departments. Each of these departments is headed by a senior management official who reports directly to the Comptroller of the Currency, the head of the agency. While the organizations and staff of each of these departments interact, they each have their own mission and goals and function somewhat autonomously. “Economics” is one of these departments.

1.1.2 OCC Economics

The primary mission of Economics within the OCC is to deliver economic and quantitative analysis to policymakers and bank examiners. OCC Economics supports on-site and off-site supervision of banks, provides current economic analysis, supports policy development, and conducts original research to support the OCC mission of ensuring a safe and sound national banking system.

Examples of activities performed by OCC Economics include:

· Examination of banks’ use of financial models

· Fair-lending screens and fair-lending exams

· Examiner training and industry outreach

· Research on current and emerging issues in banking, finance, and economics

· Participation in international and interagency working groups

· Analysis and briefings focused on assessment of risks in the aggregate, both domestic and international

· Interagency policy development and coordination

· Economic impact analyses.

To fulfill these responsibilities, OCC Economics employs a team of economists, mathematicians, statisticians, financial and policy analysts, and technology professionals to perform highly complex, resource-intensive data processing and financial modeling. Given the nation’s dynamic banking system and its increasing exposure to international developments, the assessment of risk in the banking system and the associated responsibilities for economic analysis require sophisticated data processing, data storage, and computing systems. Moreover, to keep pace with changes in the environment, these resources must (a) expand or contract as necessary; (b) support a diverse set of needs and applications; and (c) be maintained, integrated, organized, and secured to ensure OCC Economics is able to fulfill its current mission as well as meet increasingly complex challenges.

OCC Economics has outgrown its current computing infrastructure, which consists of hundreds of individual workstations, four servers, and highly distributed storage resources. As data volumes and financial modeling requirements continue to expand, this infrastructure is failing to meet the needs of OCC Economics. In addition, the current infrastructure poses increasing risks of several types, including duplication of large data sets and lack of conformance with information technology (IT) security requirements.

1.1.3 The Economics Computing Modernization Project

The OCC has initiated the Economics Computing Modernization Project (ECMP) to address emerging requirements that today’s infrastructure environment cannot support. In mid-2009, the OCC initiated the project with OCC Information Technology Services (ITS); within ITS, the Business Relations and Project Management Office (known as the “PMO”) manages the project.

The primary strategic objective of ECMP is to create an infrastructure for OCC Economics’ data analytics applications and associated management systems that will mitigate computing risks, provide a scalable approach to OCC Economics’ rapidly expanding data and application requirements, and support OCC Economics’ mission. Specifically, the project will:

· Identify a highly available, redundant, externally hosted solution that ensures high performance even for mobile users. It will support all current applications without unwanted data or hardware redundancy.

· Implement a scalable solution to accommodate large and/or unexpected requirements for additional data retention and analysis capacity, as well as additional users and applications.

· Utilize a security strategy that ensures that sensitive data—including personally identifiable information (PII)—is adequately protected and that all data is backed up and retrievable on a pre-defined schedule.

The scope of ECMP is limited to the technical solution that meets the requirements described in the ECMP Technical and Performance Requirements. Improvements or changes to OCC Economics’ business processes not necessitated by the solution are not in scope. Rationalization or standardization of the OCC Economics’ tools and data sets is also not in scope, except to the extent that the solution may provide OCC Economics with new techniques to manage data and applications.

1.2 Document Purpose

This Statement of Objectives (SOO) defines the scope of work and the expectations for the desired solution being sought from Offerors through the ECMP Request for Proposal (RFP). It outlines the objectives of ECMP, the required deliverables and performance measures, and the constraints under which Offerors must submit proposals and conduct business at the OCC.

The RFP and its attachments contain the expectations for ECMP and invite qualified solution providers to propose and price a solution that meets the specifications detailed in the ECMP Solution Use Case Model and the ECMP Technical and Performance Requirements, both appendices of the RFP. Proposals will conform to the Instructions to Offerors included in the RFP, and they will be used as input into any resulting contracts. This SOO is also an attachment to the RFP.

The intention of utilizing an SOO instead of a conventional performance work statement is to provide some degree of flexibility for each Offeror to propose an innovative, best-value solution. Offerors shall use the SOO with other applicable portions of the RFP as the basis for preparing their proposals. Offerors shall ensure that all aspects of the SOO are addressed in their proposals.

2 Solution Objectives

2.1 Introduction

The objective of ECMP is to implement a highly available, redundant, externally hosted solution that ensures high performance, provides scalability of computing capacity and storage, and utilizes a robust security strategy to adequately protect and back up sensitive data (including PII). The solution will support the OCC Economics business processes, including both operational and ad hoc analysis, as well as operational support processes.

The following benefits are expected as a direct result of the implementation of the Economics Computing Modernization (ECM) solution:

· Increased efficiency of extraction, transformation, and load (ETL) processes

· Increased efficiency of analytical processes

· Improved data and metadata management

· Increased efficiency in storage usage and allocation

· Reduced business interruption from unanticipated downtime

· Increased data security, availability, and recoverability

· Reduced lag time to scale infrastructure for increased capacity

· Increased solution monitoring and reporting capabilities.

2.2 Implementation Objectives

2.2.1 Phases and Stages of Implementation

Figure 2–1 Overview of ECM Proposed Implementation Schedule

The OCC requires a phased implementation of ECMP to address concerns associated with the perceived technical complexity of the solution and the attendant challenges of change management. Specifically, the OCC believes that a phased approach to implementation will help it achieve the following objectives:

· Successful deployment of a core subset of capabilities before investing additional resources in the implementation

· Demonstration of the viability of the overall solution based on the successful implementation of this subset of capabilities

· Facilitation of the adoption of the solution within the Economics user community.

Given its desire to mitigate the risks that could prevent the achievement of these objectives, the OCC anticipates the delivery of the solution in four phases. Nevertheless, the OCC will consider proposals that recommend a different phasing strategy, including a different number of phases and/or a different scoping of the work within each phase. Offerors proposing a different phasing strategy are asked to explain the proposed strategy in terms of its impact on the following factors: the technical complexity of the solution; the cost of the solution; the project schedule; and the ability of the proposed phasing strategy to mitigate technical, cost, and/or schedule risk.

Offerors who propose a different phasing strategy should be advised that in terms of OCC mission requirements, the timely deployment of functionality to support the analysis of retail data is a critical need. For this reason, the OCC has assigned to Phase I the deployment of retail data for use with SAS applications.

The OCC wants to pilot the ECMP solution to demonstrate the successful implementation of selected aspects of user functionality within the solution environment. Table 2–1 defines the scope of the pilot in terms of its estimated timeframe, the functionality to be piloted, the users who will participate, and the applications and data sets to be supported. If Offerors wish to propose changes to the scope of the pilot, they may do so as long as they justify these changes in terms of their ability to reduce the technical complexity of the implementation, decrease project costs, shorten the project schedule, and/or mitigate risk.

Please note that the OCC has budgeted between $1.5 million and $2 million to complete the pilot. Offerors who cannot complete the pilot within this budget should explain the constraints and/or assumptions that believe necessitate the expenditure of additional funds to complete the pilot successfully. The ability to leverage aspects of the pilot infrastructure to support future phases of the implementation will be viewed favorably. Such leveragibility will be considered when evaluating proposals whose pricing of the pilot does not fall within the budget specified by the OCC.

Table 2–1 Definition of the Scope of the ECMP Pilot

· Document lessons learned

· Make go/no-go decision for subsequent phase of the solution

· Initiate award of the Phase I option if “go” decision is made*

· Determine if any change to approach for subsequent phase(s) is required

· Transform and load data, assign minimal metadata to data sets, promote data sets into different environments, and move data in/out of the pilot environment

· Analyze data and conduct quality assurance (QA) on the data

· Perform modeling

· Perform basic system monitoring N/A

N/A

Assuming the pilot is successful, the OCC will award the option to perform the first phase of the implementation. The first phase will deliver a solution capable of meeting all Phase-1 use cases, as defined in the ECMP Solution Use Case Model, and Phase-1 requirements, as defined in the ECMP Technical and Performance Requirements. In this initial phase, users will connect to the solution’s infrastructure through the OCC network from their OCC desktops using SAS client software. Phase 1 will include:

· Acquisition (via file transfer protocol [FTP], web, and external media), conversion, migration and availability of all retail data (286 TB)

· Setup and configuration of 50 users, including system administrators, data managers, and retail data analysts

· Installation and configuration of all SAS applications (server-based)

· Conversion, migration and availability of all retail ETL programs

· Support of all retail data analysis.

The second phase will deliver an expanded solution capable of meeting all Phase-2 requirements. In this phase, the users will establish a connection (via virtual private network [VPN] or some other form of connection utility) from their OCC desktops to their virtual desktops that will be part of the solution. This connection supplements and does not replace the connections established in Phase 1. Phase 2 will include:

· The implementation of virtual machines for all users involved in Phase 1.

· Installation and configuration of all applications (client and server-based) used for processing retail data by all users involved in Phase 1.

The third and final phase is an expansion of the first two phases to cover all data (an additional 23 TB), all users (an additional 100), and all applications (retail applications for the additional users and non-retail applications for all users). It will deliver a solution capable of meeting all remaining use cases and requirements. Specifically, it will include:

· Acquisition (via FTP, web, and external media), conversion, migration and availability of all non-retail data (23 TBs)

· Conversion, migration and availability of all non-retail ETL programs

· Establishing external database connections

· Establishing the connection to the OCC operational data store (ODS) and support all associated functionality

· Incorporating the data applications that interface with the SAS INTRNET web server and use SAS Component Language (SCL) from SAS/AF.

Each of the three phases includes two stages. The first stage, implementation, will include all work and deliverables required to plan, design, build, test, and deliver the solution.

This will include migration of required data and of the ETL programs. In addition, migration of the necessary analytical programs to support user acceptance testing will be required. Please note that with respect to “required” data, only primary (i.e., original) data sets will be migrated into the new database structure, assuming that a database is proposed as part of the solution. Derivative data sets (created by OCC staff from original data sets and stored as SAS files) will be copied by the OCC users from the current environment into either the user or the collaboration environments within the ECMP solution. These derivative data sets do not need to be converted into a new data structure and migrated by the Offeror into a database.

After the implementation period is complete, the operations and maintenance (O&M) activities of the second stage will begin as detailed in the Solution Support requirements and in the service level agreement (SLA). During the first three months of the O&M stage, a period of transition will take place in which all research and analytical programs not converted during the implementation will be converted and migrated to the solution.

In proposing staff and scheduling the transition, Offerors should anticipate that two types of support will be required:

· The solution-provider’s staff will be responsible for converting and migrating the research and analytic programs not converted during the implementation stage (because they were not needed to support user-acceptance testing).

· The solution-provider’s staff will be responsible for conducting informal training and knowledge-sharing with the OCC staff to instruct them on how to write and execute the scripts used to migrate the programs from the “as is” to the “to be” environment.

In addition to the transitioning of the research and analytical programs, the O&M period of each implementation phase requires the Offeror to provide the services identified in Section 2.4 of this SOO. For more detail on expectations regarding these services, Offerors should consult the ECMP Technical and Performance Requirements, included in Appendix E of the RFP.

2.2.2 Success Criteria for the Implementation Phases

The success criteria detailed in Table 2–2 address the factors that the OCC will consider prior to proceeding to the next phase of the implementation. Failure to achieve the defined success criteria for the pilot will result in the Phase-I option not being exercised. Similarly, failure to achieve the defined success criteria for Phase I will result in Phase II not being awarded. Failure to achieve the defined success criteria for Phase II will result in Phase III not being awarded. No success criteria are provided for Phase III since there are no additional options to be exercised after Phase III for which a go/no-go decision needs to be made.

Table 2–2 ECMP Phase Success Criteria

· The subset of Retail SAS data required to test the use cases was successfully migrated to the new infrastructure.

· The pre-determined subset of programs required to test the use cases was successfully migrated to the new infrastructure.

· All aspects of vendor support were satisfactory to the extent required for the success of the pilot.

· All pilot users were connected to the solution, had access to all the migrated data (to which they had permissions), and could utilize SAS, Matlab, Stata, and MS Office.

· The cost of the pilot remained within budget.

· The implementation timeline for the pilot remained on schedule.

· The required level of throughput for the SAS applications was achieved or to the extent it was not achieved during the pilot, the solution provider successfully demonstrated that the performance issues could be resolved satisfactorily during Phase I.

· All required security features implemented during the pilot were successfully operated and maintained.

· All deliverables were delivered in a timely manner and met the quality-assurance and quality-control requirements specified in the contract.

· All Retail SAS data was successfully migrated to the new infrastructure.

· All aspects of vendor support met or exceeded the expected levels defined in the SLAs.

· The cost of Phase I remained within budget.

· The implementation timeline for Phase I remained on schedule.

· All 60 Retail SAS users were connected to the solution, have access to all the migrated data (to which they have permissions), and can utilize all the SAS applications (to which they have permissions).

· The pre-determined subset of SAS programs was successfully migrated to the new infrastructure.

· The required level of throughput for the SAS applications was achieved.

· All required security features implemented during Phase I were successfully operated and maintained.

· All deliverables were delivered in a timely manner and met the quality-assurance and quality-control requirements specified in the contract.

· All aspects of vendor support met or exceeded the expected levels defined in the SLAs.

· The cost of Phase II remained within budget.

· The implementation timeline for Phase II remained on schedule.

· All 50 Retail SAS users are connected (via VM) to the solution, have access to all the migrated data (to which they have permissions), and can utilize both the SAS and the virtualized applications (to which they have permissions).

· The pre-determined subset of programs was successfully migrated to the new infrastructure.

· The required level of throughput for the virtual applications was achieved.

· All required security features implemented during Phase II were successfully operated and maintained.

· All deliverables were delivered in a timely manner and met the quality-assurance and quality-control requirements specified in the contract.

2.3 Infrastructure Objectives

The solution infrastructure will support all OCC Economics business processes, including both operational and ad hoc analysis, as well as operational processes. It will support all current users, data, and applications with the ability to scale to accommodate additional data, applications, and users.

In the Technical and Performance Requirements appended to this RFP, the OCC has provided a list of the current applications in use by the OCC and has designated which are server-based, client-based on virtual machines, or both. (Please see Appendix A of the Technical and Performance Requirements, entitled Tool Classification Scheme.) The OCC expects that Offerors will propose the most cost-effective and efficient distribution of applications across servers and virtual machines. Wherever technically feasible and cost-effective, the OCC prefers that applications in the solution to be server-based. To the extent that Offerors depart from this model, they shall explain the rationale for their variations.

Offerors should be aware that the OCC will retain its current set of laptop computers and the analytic applications on those laptops for Economics staff. This infrastructure will be used by staff who are disconnected from the OCC network (e.g., when working remotely on a bank examination). Offerors will not be expected to maintain or support these laptops within the scope of the ECMP solution.

Please note that only those applications listed in the Tool Classification Scheme are required to be included in the ECMP solution. If a given application is not specified by that tool scheme, Offerors should consider it to be neither required nor prohibited by the OCC. It therefore may be proposed if Offerors can justify its inclusion on the basis of its technical merits and its ability to contribute to a best-value solution.

2.4 Support-Services Objectives

The solution shall include the following services as defined in the ECMP Technical and Performance Requirements:

· Change Management Services

· Configuration Management Services

· Solution Monitoring and Notification Services

· Environment Management Services

· Hardware and Software Management Services

· Network Management Services

· Data and Metadata Management Services

· Database Management Services

· Security And Control Services

· Capacity Planning Services

· Help Desk Services

· Code Conversion, Parallelization, and/or Optimization Services, including training and knowledge transfer to support OCC data managers in performing the migration of data sets from the “as is” to the “to be” environment

· SLA Management Services

· Risk Management Services

· License Management Services

· Disaster Recovery (DR) Services.

Please note that some of these services exist today (e.g., help desk services) and are provided either by contractors or by internal OCC staff. Other services do not exist today in the OCC environment (e.g., data and metadata management services). In situations where a service may exist today, the OCC is not requesting that the processes and/or tools used to support the existing services remain in place. (As a result, the OCC is not providing information on the current tool set.) The OCC requests that Offerors propose whatever processes and automated tools they believe are most appropriate to support both a best-value solution and the implementation strategy that they are proposing.

If an Offeror deems any of these services unnecessary to support its proposed solution, that Offeror may omit them from its solution, but it should explain its rationale. Similarly, if additional services are required to support an Offeror’s solution, that Offeror may propose them; however, it must provide a cogent justification of the technical, management, and/or business reasons for the OCC to incur the additional costs associated with these services.

2.5 Solution SLA Objectives

The OCC anticipates that the SLA mutually agreed to by the Offeror and the OCC will include performance measures that provide the OCC with the ability to meet the objectives identified in Table 2–3. These objectives are associated with one or more factors deemed by the OCC to be critical to the success of ECMP. Mindful of these objectives, the OCC has created an SLA (provided in Appendix C of this RFP) to describe what it believes are meaningful performance measures associated with the ECMP solution. Using this SLA as a baseline, Offerors are free either to change existing measures or propose additional performance measures. When proposing additional SLA performance measures, Offerors should identify the OCC objectives (from Table 2–3) that are addressed by these additional measures. Please see Section 4.4 of this SOO for a discussion of the disincentives and incentives associated with the SLA.

Table 2–3 ECMP SLA Objectives

· Ensure that all implemented changes address the problem or situation for which they were requested

are implemented in the mutually agreed timeframes

· Ensure that all deployed releases contain all changes—and only those changes—planned for the release

· Ensure the success of releases by eliminating rollbacks due to failure

· Ensure that the OCC is notified when resource utilization reaches 75% of capacity in the following areas:

· Central Processing Units (CPUs)

· Memory

· Data-storage utilization

· Network Utilization

· Ensure that the uptime and network connectivity of all graphical user interfaces (GUI) meet OCC’s requirements

· Disk input/output (I/O), including throughput

· Network performance, including throughput (NOTE: Network service levels apply to the solution provider's network only. Errors on the OCC network or other external networks are excluded.)

· Database performance

· Ensure timely installation of security patches and anti-virus updates issues detected via monitoring

· Ensure timely resolution of environmental issues issues detected via monitoring

· Ensure response times to user-initiated help desk calls meet requirements

· Ensure response times to user-logged help desk tickets meet requirements

· Ensure help desk resolution times meet requirements

· Ensure incident resolution meets OCC customer satisfaction expectations, including those for timeliness and quality

· Ensure DR content recoverability meets requirements

To ensure the OCC has visibility into the SLAs and the ability to manage them appropriately, the OCC has requested a series of compliance reports, notifications, alerts, and reviews that will provide visibility of the extent to which individual service levels have (or have not) been achieved. These notification and reporting artifacts, as documented in either the SLA itself or the ECMP Technical and Performance Requirements, are necessary for the OCC to obtain insight into and understanding of the solution provider’s ability to deliver the proposed solution successfully.

Please note that the frequency of these reporting and notification mechanisms, as documented in the SLA and the requirements, is (in some places) described as “negotiable with regard to timeframe.” (See the ECMP Technical and Performance Requirements, which identifies the requirements that are negotiable and in what regard.) The OCC wishes the Offeror to understand that to the extent a report or notification can be delivered with the requested frequency without a code customization being required, the OCC wishes that specific timeframe to be honored. To the extent that a code customization is required to deliver a given report or notification with a specified frequency, the OCC is open to negotiating the timeframe for that delivery.

For example, if the OCC has requested a real-time notification, and such a notification can be generated without any code customization, the OCC asks that the real-time notification be provided. If custom code must be written to generate a real-time notification, but a notification can be generated one hour after the triggering event without any customization of code, the OCC would be willing to consider the “out of the box” notification as meeting its requirements. The OCC appreciates that reducing the level of customization in the system-monitoring tool(s) not only reduces implementation cost but reduces the level of technical risk associated with the ECMP solution.

2.6 Solution Testing Objectives

Testing will be comprised of system, regression, 508 compliance, and user acceptance testing (UAT); these types of testing will be conducted in each phase, to the extent required, as outlined below. OCC expects the test environment(s) to be provided by the Offeror as part of the solution.

2.6.1 Pilot Tests

2.6.1.1 System Testing

System testing will be conducted by the solution provider and will include the following tests:

· Functional testing to ensure all pilot use cases, operational requirements, data management requirements, and audit requirements are met

· Component and integration testing to ensure the technical requirements associated with the pilot are met

· Performance testing to ensure performance and capacity requirements are met . To the extent that performance requirements are not met during the pilot, testing must demonstrate—no later than the completion of the pilot—how performance requirements will be successfully met during Phase I.

· Availability testing to ensure the availability requirements associated with the pilot are met

· Security testing to ensure the security requirements associated with the pilot are met

2.6.1.2 User Acceptance Testing

UAT will be conducted by the OCC users and will include the following system tests:

· Functional testing of all pilot use cases (using retail data with SAS applications) and VM software (including MS Office, Stata, and Matlab)

· Performance testing to ensure performance and capacity requirements are met . To the extent that performance requirements are not met during the pilot, testing must demonstrate—no later than the completion of the pilot—how performance requirements will be successfully met during Phase I.

· Availability testing to ensure the availability requirements are met

· Security testing to ensure the security requirements are met

2.6.2 Phase-I Tests

2.6.2.1 System Testing

System testing will be conducted by the solution provider and will include the following tests:

· Functional testing to ensure all use cases, operational requirements, data management requirements, and audit requirements are met

· Component and integration testing to ensure the technical requirements are met

· Performance and load testing to ensure performance and capacity requirements are met and to establish the maximum load of the solution architecture

· Availability testing to ensure the availability requirements are met

· Failover and recovery testing to ensure the recoverability requirements are met

· Security testing to ensure the security requirements are met.

2.6.2.2 508 Compliance Testing

To the extent necessary, 508 compliance testing will be facilitated by the solution provider and be executed by the OCC to ensure the IT accessibility requirements are met.

2.6.2.3 User Acceptance Testing

UAT will be conducted by the OCC users and will include the following system tests:

· Performance and load testing to ensure performance and capacity requirements are met and to establish the maximum load of the solution architecture

· Availability testing to ensure the availability requirements are met

· Failover and recovery testing to ensure the recoverability requirements are met

· Security testing to ensure the security requirements are met

· Functional testing of all Phase-1 use cases using retail data with SAS applications.

2.6.3 Phase-II Tests

2.6.3.1 System Testing

System testing will be conducted by the solution provider and will include the following tests:

· Functional testing to ensure all use cases, operational requirements, data management requirements, and audit requirements are met

· Component and integration testing to ensure the technical requirements are met

· Performance and load testing to ensure performance and capacity requirements are met and to establish the maximum load of the solution architecture

· Availability testing to ensure the availability requirements are met

· Failover and recovery testing to ensure the recoverability requirements are met

· Security testing to ensure the security requirements are met.

2.6.3.2 Regression Testing

All system tests from Phase I will be re-run by the solution provider.

2.6.3.3 508 Compliance Testing

To the extent necessary, 508 compliance testing will be facilitated by the solution provider and be executed by the OCC to ensure the IT accessibility requirements are met.

2.6.3.4 User Acceptance Testing

UAT will include the following tests:

· Functional testing of the following use cases using retail data with all applications identified for the ECM solution as retail data applications

· Conduct Analysis

· Store Analysis Output

· Transform Data Files

· Establish External Database Connection

· Administer User Access

· Execute Data Request

· Execute Storage Request

· Execute Software Request

· Monitor Solution

· System Log-in/Log-out.

2.6.4 Phase-III Tests

2.6.4.1 System Testing

System testing will be conducted by the solution provider and will include the following tests:

· Functional testing to ensure all use cases, operational requirements, data management requirements, and audit requirements are met

· Component and integration testing to ensure the technical requirements are met

· Performance and load testing to ensure performance and capacity requirements are met and to establish the maximum load of the solution architecture

· Availability testing to ensure the availability requirements are met

· Failover and recovery testing to ensure the recoverability requirements are met

· Security testing to ensure the security requirements.

2.6.4.2 Regression Testing

All system tests from phases 1 and 2 will be re-run by the solution provider.

2.6.4.3 508 Compliance Testing

To the extent necessary, 508 compliance testing will be facilitated by the solution provider and be executed by the OCC to ensure the IT accessibility requirements are met.

2.6.4.4 User Acceptance Testing

UAT will include the following tests:

· Functional Testing of the following use cases using non-retail data sets with all applications identified for the ECM solution as non-retail applications

· Conduct Analysis

· Store Analysis Output

· Transform Data Files

· Establish External Database Connection

· Upload Data from OCC Network Drives

· Upload Data from ODS

· Create Data Files from ODS

· Establish External Database Connection

· Monitor Solution.

In addition, functional testing will be required to:

· Test the web connections to the commercial data applications

· Test the data applications that interface with the SAS INTRNET web server and use SCL from SAS/AF.

3 Deliverables and Deliverable Schedule

3.1 Submission of Deliverables

For purposes of formal delivery, all deliverables shall be provided by e-mail to the Contracting Officer’s Technical Representative (COTR), the Contracting Officer (CO), and the Contract Specialist (CS).

3.2 Deliverable Types

Understanding the types of deliverables required under this SOO, the COTR’s process for deliverable inspection and acceptance, and the document review process are essential to understanding deliverable requirements and the standards for deliverable acceptance. Descriptions of the inspection process and acceptance criteria follow.

3.3 The PMO Document Review Process

The PMO document review process is one of the PMO’s quality assurance processes. The process is intended to reduce the risk that poor-quality documents are delivered to the PMO’s customers.

Only some documents undergo the document review process. For practical purposes, some of the administrative documents, like weekly status reports, do not undergo the document review process. Table 3–32 through Table 3–35 specify the documents that will undergo the document review process.

The contractor must produce four distinct versions of a document that undergoes the document review process:

1. The management review version

2. The quality assurance review version

3. The PMO-customer review version

4. The final version.

Each version is a contract deliverable. Each version has a contract due date, and failure to deliver any version on time results in a missed timeliness milestone, which impacts the monthly timeliness performance measure.

The document review process proceeds as follows. The contractor delivers the management review version of the document. The management review version is reviewed by the PMO Project Manager and the PMO Project Manager Team Lead. (Since one of these individuals is designated as the contract COTR, the management review includes, by default, the COTR’s initial review of the deliverable.) After receiving the OCC’s comments on the management review version, the contractor will:

1. Consolidate comments received from the reviewers

2. Review and suggest a resolution for each comment

3. As necessary, work with the reviewers to resolve comments

4. Obtain the COTR’s approval for all comment resolutions

5. Revise the document to produce the next version, in this case the quality assurance review version.

The quality assurance review version is then reviewed by two members of the PMO who are not part of the project team. After receiving the OCC’s comments, the contractor then repeats steps 1 through 5 to produce the PMO-customer review version. The PMO-customer review version is then reviewed by team members in OCC Economics. As necessary, this version may also be reviewed in parallel by selected members of ITS (e.g., the security or network subject-matter experts). After receiving the OCC’s comments, the contractor repeats steps 1 through 5 to produce the final version of the document.

The COTR is ultimately responsible for the inspection and acceptance of deliverables. This responsibility is delegated to the COTR by the Contracting Officer. Any disputes will be resolved by the Contracting Officer.

In addition to comments, the COTR will assign a deliverable quality score to each deliverable, in accordance with this contract. The COTR-assigned score will be used to calculate the monthly deliverable quality performance metric for the contractor. Any deliverable with a COTR-assigned score of 3 or higher is presumed accepted. Any deliverable with a score less than 3 is unacceptable and will be rejected by the COTR.

The OCC fully intends to conduct inspections as described in this section, and the contractor should assume this process for cost estimating, planning, and proposal development purposes. However, this section shall not in any way limit the OCC’s inspection rights set forth in Federal Acquisition Regulation (FAR) 52.246-4 and 52.246-6, and the OCC reserves the right to modify the inspection process should it become necessary during the performance of the contract.

3.3.1 Review Times

Table 3‑1 lists the minimum time the OCC has to review each version and deliver comments to the contractor. The contractor should use these minimums for schedule planning purposes. In some cases, the OCC may require longer review times. On fixed-price tasks, if the minimum times were assumed in the schedule and the OCC required longer review times, the contractor should request an equitable adjustment if it can demonstrate that longer review times resulted in a material impact to the cost of completing the task.

Table 3‑1 also lists the maximum time the contractor has to revise each version after receipt of comments. The contractor should use these maximums for schedule planning purposes. Requests for additional revision time should be made to the COTR and will be evaluated on a case-by-case basis. That said, contractors must realize that formal contractual due dates can only be changed by the Contracting Officer and a request to the COTR is no guarantee that a date change will be granted by the CO.

Table 3‑1 Minimum Times for Deliverable Review and Revision (Phases I – III)

Management Review Version

During the pilot phase of the implementation, the timeframe for deliverable review and revision will be shortened to accommodate the compressed schedule. Management reviews will occur in three (3) business days. Quality-assurance and customer reviews will run in parallel (rather than sequentially) and will be allotted five (5) business days for completion.

Formal contractual time minimums notwithstanding, when practical, the OCC will make efforts to minimize the total review time. Experience has shown that the OCC can sometimes deliver comments faster than the minimum review time. Also, under certain circumstances, given agreement between the COTR and the contractor, the process can be modified to expedite review. For example, the management review and quality assurance review might be conducted in parallel using one version of the document. However, for schedule planning purposes, the contractor should assume the minimum review times, unless discussed in advance with the COTR.

3.3.2 Number of Reviewers

Generally speaking, there are two reviewers for the management review version and two for the quality assurance review version, although the OCC may add reviewers in situations where their feedback is expected to be value-added.

The number of reviewers varies more widely for the PMO-customer review version, depending on factors like the number of project team members from the customer business unit and whether ITS subject matter experts (SME) are also reviewing the document. However, given the mature state of the PMO-customer review version, experience has shown that the larger number of reviewers generally does not result in a material increase in the amount of effort to review comments and produce the final version. In most cases, the PMO-customer review version will include comments from no more than six (6) individuals from OCC Economics and two (2) ITS SMEs.

3.3.3 Review Guidelines and Scores

For simplicity and consistency, reviewers use the same guidelines to assess a narrative document that the COTR uses and will provide a score for the narrative document. Reviewer scores are advisory and are not used to calculate the monthly deliverable quality metric; only the COTR score is used for that purpose. However, the COTR may use reviewer scores as advisory input to his/her score.

3.3.4 Version Requirements for Documents Not Being Scored

The contractor must produce two distinct versions of a document that does not undergo the extensive document review process described in Section 3.3 (and thus is not scored):

1. The management review version

2. The final version.

For these documents, if the COTR does not require any changes after inspection, the management review version automatically becomes the final version. Otherwise, the contractor will incorporate the requested changes to produce a new (and final) version.

3.4 Importance of the Management Review Version

While the management review version is one of three versions produced before the final version of a deliverable, it should be a complete, polished document, not a working draft. In the course of developing a deliverable, the contractor should plan for as many working drafts and informal reviews as necessary before delivering the management review version. (These informal reviews should include a discussion of an annotated outline of the proposed deliverable with the COTR and Project Manager in advance of creating any content.) The document review process is a quality check at the end of the document development process; it is not a time to be collecting facts, creating content, working through composition issues, or fixing document formatting, grammar, or editorial style. Understanding this expectation will make the document review process proceed quickly and increase the chance of acceptance of deliverables by the COTR.

3.5 Remediation of Unacceptable Deliverables

The OCC will require the contractor to re-work and re-submit unacceptable deliverables until they receive a satisfactory score. For the purposes of assessing the timeliness of a deliverable, unacceptable deliverables will also be considered late if they are judged by the COTR, upon submission, to be incomplete or are of such poor quality that they are essentially incomplete.

Consistent with FAR 52.246-4, the contractor will correct deliverables produced under fixed-price tasks without additional cost to the OCC. Consistent with FAR 52.246-6, the contractor will correct deliverables produced under time-and-material or labor-hour tasks at a cost to the OCC that excludes contractor profit.

3.6 Required Deliverables

Table 3–2 through Table 3–5 list deliverables to be included by the Offeror in its technical approach.