ESBD_File_220222_21-0000096_Solicitation_Final.pdf

Teacher Retirement System of Texas

Request for Proposal (RFP) TEAM Performance Testing Services

Solicitation No.: 21-0000096 NIGP Class/Item Codes: 920-76

Anticipated Schedule of Events

11/20/2020 Issuance of RFP

12/4/2020, 4:00 p.m., Central time Non-Mandatory Pre-Proposal Conference.

Pre-Proposal Conference will be held via Microsoft Teams Meeting. Conference call information provided in Item 14 below.

12/8/2020, 3:00 p.m., Central time Deadline for Submission of Questions.

12/21/2020, 3:00 p.m., Central time Deadline for Submission of Proposals.

Late Proposals will not be considered

TBD Presentations by Top-Ranked Respondents, if requested. TRS may elect to conduct presentations (or interviews) as part of the evaluation process.

TEAM Performance Testing Services Solicitation No.: 21-0000096

Table of Contents I. Summary

II. Definitions and Acronyms

III. Scope of Work

A. Introduction

B. Program Overview

Performance Test Service Level Expectations

IV. Evaluation Criteria

A. Minimum Requirements/Qualifications

B. Statement of Qualifications and Experience – Selection Criteria: 15 %

C. Respondent’s Key Personnel Qualifications and Experience – Selection Criteria: 35 %

D. Technical Approach – Selection Criteria: 30 %

E. Pricing – Selection Criteria: 20 %

F. Oral Presentations/Interviews (Optional) – Selection Criteria: 20 points

V. Price Form

VI. HUB Subcontracting Plan Information

A. Agency Annual HUB Goals:

B. HUB Subcontracting Opportunities for this Solicitation

C. Finding HUB Vendors

VII. Insurance Section

A. COVERAGES REQUIRED

B. POLICIES

C. RENEWAL CERTIFICATES

D. NOTICES OF CHANGE

E. INSURANCE CERTIFICATE

F. REQUIRED ADDITIONAL PROVISIONS

VIII. NOTICES, REPRESENTATIVES, AND RECORDS LOCATION

IX. Proposal Submission Format Requirements

A. Proposal Format

B. Submission Instructions

X. Proposal Content

XI. Respondent’s Signature and Information

XII. Acknowledgement of Additional Documents

XIII. Exception Language

XIV. The Impact of Senate Bill 65 on TRS’s Contracting Practices

XV. Contract Document List

TEAM Performance Testing Services

I. Summary

1. Purpose/Overview Teacher Retirement System of Texas (TRS) is seeking a Contractor to develop, execute, and analyze a performance testing tool for an internally developed web self-service application. TRS is also seeking a knowledge transfer pertaining to the performance testing tool the Contractor proposes.

2. Type of Solicitation Request for Proposal (RFP)

3. Issuing Office

Procurement & Contracts 1000 Red River Street Austin, TX 78701-2698 ATTN: Aundre Petty

4. Purchasing Authority Competitive Procurement per Texas Government Code Section 825.103

5. Minimum Requirements / Qualifications

Proposals that do not meet the following minimum requirements will be rejected as non-responsive and disqualified:

Minimum Requirements:

1) See page 20 – Section IV. Evaluation Criteria

(A) Minimum Requirements

6. Responses to Solicitation Competitive Proposals

7. Deadline for Proposals Email of Proposal Submission must be received by Teacher Retirement System of Texas (TRS) no later than: 12/21/2020, 3:00 p.m., Central time

8. Initial Contract Term 3/1/2021 – 2/28/2023

9. TRS Optional Renewal Terms No optional renewal periods.

10. TRS Procurement and Contracts Representative for this Solicitation

Aundre Petty Aundre.Petty@trs.texas.gov

11. TRS Historically Underutilized Business (HUB) Coordinator

LaTresa Stroud

(512) 542-6307 LaTresa.Stroud@trs.texas.gov

TEAM Performance Testing Services

12. Proposals Submitted by Email Accepted – Submit the response, via email only, to the TRS Procurement and Contracts Representative listed for this Solicitation specified in Item 10.

13. Proposals Submitted by Fax Not Accepted

14. Pre-Proposal Conference 12/4/2020, 4:00 p.m., Central time via Microsoft Teams Meeting. To receive a Microsoft Teams Meeting invite, Respondents must email the TRS Procurement and Contracts Representative for this Solicitation, specified in Item 10, no later than 4:00 p.m., Central Time, 12/2/2020.

15. Solicitation Addenda Notices of changes to items directly impacting the original Solicitation or Proposal process will be posted on the Electronic State Business Daily (ESBD) located at www.txsmartbuy.com/sp. TRS will post any amendment to this solicitation on the ESBD as a Solicitation Addendum. It is the responsibility of an interested party to periodically check the ESBD for updates to the Solicitation prior to submitting a Proposal. Each Respondent is solely responsible for verifying TRS’s receipt of its questions, if applicable, and Proposal by the deadlines specified in this Solicitation.

16. Questions and Answers Questions regarding this Solicitation must be in writing and must be submitted to the TRS Contact Person specified in Item 10 no later than 12/8/2020, 3:00 p.m., Central time. Telephone inquiries will not be accepted. Questions must be submitted by email to the email address specified in Item 10.

17. Respondent Presentations / Interview

TRS may request that Top-Scoring Respondents present an overview of the Services proposed in their Proposals. The Presentations/Interview will be performed virtually via Microsoft Teams Meeting. The number of Top-Scoring Respondents will be determined by TRS at its sole discretion.

18. Contact with TRS Staff Upon issuance of this Solicitation, employees and representatives of TRS, other than the TRS Contact Person identified in Item 10, will not discuss the contents of this Solicitation with any Respondent, Potential Respondent, or its representatives.

TEAM Performance Testing Services

Failure of a Respondent or any of its representatives to observe this restriction may result in disqualification of any related Proposal.

This restriction does not preclude discussions between affected parties for the purpose of conducting business unrelated to this procurement.

II. Definitions and Acronyms

As used in this Solicitation, the following capitalized terms shall have the meaning specified below.

Term Definition Agency Teacher Retirement System of Texas Agency Primary Contact The person designated by TRS to be the point of contact between TRS and Contractor for the purposes of monitoring Contractor performance and providing day-to-day interaction and instruction. Also referred to as “Contract Sponsor,” and/or “Project Manager”

Contract The definitive written agreement, if any, executed by the authorized representative of each of the parties at the end of the procurement process that formalizes the terms, provisions, covenants, and obligations, including but not limited to those contained in this Solicitation, of the respective parties to the arrangement for provision of Services.

Contractor The individual or entity deemed by TRS to be qualified to provide the goods, services or both sought by TRS pursuant to this solicitation and who is ultimately selected to carry out the requirements stipulated in this solicitation and any resulting Contract. Has the same meaning as “Vendor.”

Contract Sponsor The person designated by TRS to be the point of contact between TRS and Contractor for the purposes of monitoring Contractor performance and providing day-to-day interaction and instruction during the term of the Contract. Also referred to “Agency Primary Contact” and/or “Project Manager.”

Contractor Primary Contact The person designated by Contractor to be the point of contact between Contractor and TRS for purposes of scheduling, billing, contract management, and day-to-day management of services under this Solicitation and the resulting Contract. Also has the same meaning as “Vendor Primary Contact.”

Contract Execution The date the Contract is fully executed by all Parties.

Days Calendar days unless otherwise specified.

Deliverable Unit or increment of work required by the Contract, including such items as goods, services, phase completion, reports, or documents.

Electronic State Business Daily (ESBD)

The public-facing website in which applicable government entities post certain competitive procurement solicitations.

www.txsmartbuy.com/sp.

Financial Interest Receiving, or having a right to receive, money or other valuable property or benefits under or in connection with the actual or proposed Contract;

holding a position in a business such as a principal, officer, director, trustee, partner, employee, or the like, or holding a position of management; holding any of the outstanding debt of a person directly involved in the proposed or resulting Contract; or owning stock or other interest in the business.

Fiscal Year (FY) The State of Texas and TRS’s fiscal year that begins September 1 and ends August 31.

TEAM Performance Testing Services

Term Definition Key Personnel Individuals who contribute to the development or execution of a project in a substantive, measurable way, and whose absence from the project would be expected to impact the performance of the Contract.

Mandatory Required, compulsory, or obligatory.

Offer, Proposal, or Response The documents, including any exhibits or attachments, submitted by a prospective Respondent in response to a solicitation. “Offer” has the same meaning as “Proposal” or “Response,” when such terms are applicable in the solicitation document so titled as Request for Proposal (RFP), Request for Offers (RFO), or an Invitation for Bid (IFB), respectively.

Party Entity who enters into an agreement with one or more other contracting parties intending to be legally bound to perform the obligations for the agreed consideration.

Relative A current or former TRS employee’s spouse, father, mother, brother, sister, son or stepson, daughter or stepdaughter, mother-in-law, father-in-law, sister-in-law, brother-in-law, son-in-law, or daughter-in-law.

Respondent Each individual or entity that responds to this solicitation with a Proposal to provide the goods, services or both sought by TRS pursuant to this solicitation in accordance with the requirements stipulated in this solicitation and any resulting Contract.

Services All of the requirements, duties, schedules, locations, and other items listed in the Solicitation. The term includes any other services associated with Respondent’s performance under any Contract resulting from this Solicitation.

TRS Teacher Retirement System of Texas, the public entity responsible for the issuance of this Solicitation.

Vendor The individual or entity deemed by TRS to be qualified to provide the goods, services or both sought by TRS pursuant to this solicitation and who is ultimately selected to carry out the requirements stipulated in this solicitation and any resulting Contract. Also has the same meaning as “Contractor.”

Vendor Primary Contact The person designated by the Vendor to be the point of contact between Vendor and Agency for purposes of scheduling, billing, contract management, and day-to-day management of Services under this Solicitation. Also has the same meaning as “Contractor Primary Contact.”

Work The services, goods, and property the Contractor is required to provide in this Contract. The term includes the entire completed undertaking and the various separately identifiable parts. The term also includes all duties of Contractor.

III. Scope of Work

A. Introduction Teacher Retirement System of Texas (TRS), a public pension fund and entity of the State of Texas, issues this Request for Proposal (RFP) # 21-0000096 to solicit for a qualified Contractor to provide TRS Enterprise Application Modernization (“TEAM”) Program Performance Testing Services.

TRS is seeking a Contractor to provide performance testing services to ensure that TRS members do not experience issues once the internally developed Web Self Service (WSS) application is released into production. In addition to the Contractor developing, executing and analyzing the performance testing, TRS is also be seeking a knowledge transfer pertaining to the performance testing tool, so that TRS can better serve future needs.

B. Program Overview

i. Background

TRS is a service-oriented public pension system providing retirement, healthcare, and related benefit programs for Texas public education employees, retirees, and their beneficiaries (“Participants”). TRS’ mission is to improve the retirement security of plan participants and their beneficiaries by prudently investing and managing trust assets and delivering benefits that make a positive difference in members' lives.

TRS is undertaking an effort to modernize its core processes and technologies used to administer a range of TRS benefits. The effort, known as the TEAM Program, will modernize non-investment TRS applications. As a result of this effort, the Teacher Retirement Unified System for Technology (TRUST) system will be implemented. The TRUST system, comprising several initiatives, will utilize new technologies to improve the quality of service delivered to TRS’ customers through a new Pension Administration Line of Business (LOB) system and automated interfaces. These initiatives will involve incremental releases utilizing an Agile methodology approach.

The WSS application release for TRUST is slated to go live in Spring 2022. This will be a TRS member-facing application (myTRS). The performance tests will assess the WSS myTRS system application under heavy loads (Load Test) and WSS myTRS Portal availability during batch execution (Stress Test). The following diagram and description depict a high-level overview of the WSS architecture:

TEAM Performance Testing Services

The myTRS Portal is a J2EE, Struts-based Web application running on JBOSS. Running on clustered virtual servers, myTRS utilizes data from multiple sources (SQL Server, Legacy Mainframe, MS Dynamics CRM, third-party sources). Web service integration is handled through the JBOSS FUSE Enterprise Service Bus.

Additional information:

• Presentation layer is based on J2EE Web Container

• Uses MVC Pattern using Struts

• View is enhanced using composite view pattern to:

o Centralize page layout o Centralize Navigation menus

• Proactive Security o Buttons are disabled

• Reactive Security o Security checks on all actions

• Invokes Business Logic through a Stateless Service Provider EJB

• Responsive Web Design (RWD) design identifies the resolution of the device it is being served and sizes flexible images and fluid grids correctly to fit the device screen

• Built using Twitter Bootstrap Mobile framework. Works on most popular smart phones and tablets (including iOS & Android)

Note: The myTRS portal also uses some third-party Web services for functions like identify proofing and e-Signature. These services are not included in the Scope of Work (SOW).

TRS is utilizing an Agile approach for the WSS application release. This approach allows for continuous development and delivery using smaller, more frequent releases, business unit involvement in product development and testing, and continuous quality improvement.

TRS has established Scrum Teams that are cohesive and work efficiently through permanent communication exchange involving high transparency in project development and implementation. These Scrum Teams are comprised of dedicated cross-functional teams of business representatives, Business Analysts, developers, and quality assurance individuals which will work together to deliver the WSS myTRS product.

TEAM Performance Testing Services

TRS has a Product Backlog with a prioritized list of the features, capabilities, and fixes needed for successful release of the WSS myTRS product. The Product Backlog is prioritized by business representatives to ensure that the most critical business needs and priority items are worked on first.

The work by the Scrum Teams to deliver the WSS myTRS product is accomplished using a time-boxed 3-week Sprint focus, during which design, build, and testing is undertaken to produce completed work features on an incremental basis. Issues that could hinder such efforts are addressed immediately during daily Scrum Team meetings. A Sprint review or retrospective is conducted at the end of each Sprint to obtain stakeholder feedback. The review or retrospective helps identify potential features that should have been implemented, defects not encountered during testing, as well as work that went well. The review or retrospective also provides opportunities for Scrum Teams to alter work plans for the next Sprint.

ii. Description of Work Utilizing an Agile approach, the Contractor will be responsible for:

1. Performance Testing Assessment and Final Performance Testing Results

• Creating a Performance Test Plan

• Creating Test Scripts for the business scenarios

• Executing multiple testing iterations against the Performance Test Plan

• Analyzing issues, logging defects and recommending initial changes to increase performance and remove performance obstacles

• Establishing a baseline for the performance metrics

• Generating initial performance reports

• Updating Test Scripts, as needed based on changes by TRS

• Executing against the Performance Test Plan after implementation of changes recommended by TRS

• Analyzing issues, logging defects and recommending changes to increase performance and remove performance obstacles

• Generating final performance reports

• Participating in Agile ceremonies for the WSS myTRS product, as appropriate

2. Knowledge Transfer

• A knowledge transfer throughout the work timeline about the Contractor-developed performance testing tool

The Contractor will create a Performance Test Plan, develop test cases, and execute initial performance testing runs to assess how the WSS application performs under heavy workload in terms of responsiveness and stability. The Contractor’s approach will include both batch and online performance testing. Issues that impact performance during initial runs will be analyzed and defects will be appropriately logged. Performance testing should achieve or exceed minimum performance benchmarks established by TRS and identify obstacles that hinder WSS application performance. Once these obstacles are no longer present, the Contractor will re-execute performance testing to verify and confirm that established benchmarks for the WSS myTRS application are met.

Performance Testing Performance tests that the Contractor will be responsible include:

• Portal Load Test – The Contractor will test performance of the myTRS application while running online scenarios under normal loads and validating results against service-level expectations. Load testing is a type of performance test where the application is tested for its performance based on normal and peak usage. The performance of an application is checked with respect to its response to the user request and its ability to respond consistently within an accepted tolerance on different user loads. Tests will measure system performance as the workload increases and check for buffer overflow, memory leaks, mismanagement of memory, load balancing problems, bandwidth requirements, capacity of the existing system, and the upper limit of all WSS application components (e.g., database, hardware, network, etc.), so that future load for the application can be anticipated. Performance tests will include executing and reporting results on endurance, spike, load, and scalability tests.

• Portal Stress Test – The Contractor will test performance of the myTRS Portal using both batch and non-batch job(s) while the system is placed under stress (e.g., while running identified online scenarios under heavy loads; running large Reporting Entity reports, etc.) to identify performance obstacles, if any, and validate performance against service level requirements.

• Failover/Recoverability Testing – TRS personnel will test whether the WSS application can return to its normal state after failure or abnormal behavior and how long it takes for the application to do so. Contractor will be required to run identified test scenarios with loads and stress among other failover/recoverability testing support duties.

Performance Test scope duties the Contractor will be responsible for include:

• Creating a Performance Test plan which identifies the testing approach, strategy, testing activities, post-test analysis and bug reporting process, regressions, and reporting methodology

• Creating test cases to assess myTRS performance for identified online and batch scenarios

• Defining test data requirements for each selected test case and parameterizing scripts with test data

• Identifying and gathering key metrics to be collected from servers for Online and Stress Tests

• Identifying pages, operations, transactions, database queries, Web Service calls, and other system components that may require modification to meet service level requirements

• Assisting TRS Development and Infrastructure teams to run batch processes that consume resources (e.g., CPU, memory, disk, network) on the Web and Database servers to determine WSS portal performance during batch execution

• Executing identified online and batch scenarios

• Demonstrating that batch processing and online scenarios can be completed within agreed processing times and not be affected by peak loads

TEAM Performance Testing Services

• Analyzing, reporting, and making recommendations concerning issues identified during execution and logging defects and changes needed, as applicable

• Generating performance test result reports (initial and final assessment reports) listing results by test case

• Updating test cases and scripts, as needed, to accommodate code changes after defects are fixed

• Re-executing online and batch scenarios after all issues/defects are fixed before final report is delivered

• Creating a final performance test report, to include results and any additional changes/recommendations

High level batch and online scenarios for Portal Stress and Load testing are as follows. The online scenarios listed below must be tested during "normal business hours circumstances" and during execution of the full batch schedule to determine batch execution effect on online scenario performance. Contractor will work with TRS Infrastructure staff to coordinate the execution of all online scenarios.

High Level Scenario

Test:

Average

Load

Test:

Peak Load

(Stress Test)

Test:

Endurance (Avg load over 8-hour period) 1 myTRS Letter Generation (Batch)

2 View Member Account Home Page (Active Member)

A single web page that reads data from multiple data sources

3 View Member Account Home Page (Benefit Recipient)

A single web page that reads data from multiple data sources

4 Schedule Member Appointment

A member Web transaction that reads data from an writes data to a MS Dynamics back-end calendar via Web Service calls through and Enterprise Service Bus

5 Perform Benefit Estimate (Active Member only)

High Level Scenario

Test:

Average

Load

Test:

Peak Load

(Stress Test)

Test:

Endurance (Avg load over 8-hour period) A screen that accepts inputs from the user and performs a benefit calculation based on those inputs and database values for the member

6 Send Secure Message

A Web-based screen that allows a member to enter a message into a form that can be transmitted securely to TRS MS Dynamics system

7 Read Secure message

A screen that reads messages stored in MS Dynamics and displays them to the member

8 View Member Document Center

A Web resource that displays lists of documents for a member and retrieves a document the member selects from a disk-based file store

9 Apply for Refund (e-Sign)

An online version of TRS's Refund process that accepts member input and reads data from the database

10 Update Profile (address, e-mail, phone etc.

Updating of member profile data such as e-mail address, phone, and physical address

11 Verification of Monthly Payment (For Retirees) 12 Maintain Tax Withholding 13 Activation of myTRS Account

Performance Test Service Level Expectations SLE Activity Range SLE 1 Expected Response Time for pages 2 seconds SLE 2 Expected Response Time for pages during batch execution 4 seconds

SLE 3 Pages with large data volume and/or calculation intensive (e.g. Benefit Estimate)

4-5 seconds

SLE 4 Pages with large data volume and/or calculation intensive during batch execution (e.g. Benefit Estimate)

8-10 seconds

SLE 5 Expected number of simultaneous users (Average test load)

2,000 users

SLE 6 Expected number of simultaneous users (Stress test load)

10,000 users

SLE 7 Expected number of simultaneous users (Endurance test load)

8 hrs / 2,000 users

Note: Contractor is expected to run the online /batch scenarios with loads to support these tests when TRS is performing its assessment tests as needed.

Knowledge Transfer Plan The Contractor will be responsible for providing a Knowledge Transfer Plan for the performance testing tool the Contractor is utilizing to execute the Performance Test Plan.

The Knowledge Transfer Plan will address the following at a minimum:

• The training needed to implement and maintain the performance testing tool

• How and when the Contractor will involve TRS staff in the Contractor’s performance testing process

• The knowledge transfer process with TRS Quality Assurance (QA) staff

• The knowledge transfer process with TRS technical staff

• The training offered by role (i.e. user, administrator, developer, etc.).

• The minimum skill set needed by role, with a corresponding description of the technical skills required for TRS staff to effectively implement and maintain the tool in their respective roles

• How Contractor will validate that knowledge transfer is successful

Tool specifications TRS currently has a load testing software license from LoadRunner that allows TRS to simulate up to 5,000 virtual users. However, TRS is open to the use of other load testing software. If a Respondent proposes the use of different load testing software, the Respondent should provide information about the pricing and licensing of such software, whether it is cloud-based or on-premise, and recommend hardware requirements for an on-premise solution in its proposal.

Data Access The Contractor may have limited access to Personal Health Information in the performance of services. If such access is required, Contractor must complete the Business Associate Agreement included in the RFP.

iii. Key Personnel

Role Description myTRS Performance Test Contractor

• Responsible for the performance testing and deliverables as outlined in this SOW

• Execute performance testing activities/scenarios identified in this

SOW

• Document Performance Test plan to meet performance test objectives

• Provide weekly status reports to the QA Test Manager

• Capture and analyze results and submit regular performance test reports

• Provides testing metric reports for testing progress

• Scripts Test cases and parameterize data set up

• Escalate critical issues and log defects as needed

• Responsible for the Final Performance Test Report

• Produce deliverables identified in this SOW on schedule

Project Manager

• Coordinates schedule with the Contractor

• Coordinate communication between the Contractor and TRS staff

• Coordinate deliverables between Contractor and TRS staff

QA Manager • Reviews, finalizes, and approves test plan, test strategy, test schedule, and risk in conjunction with TRS performance teams

• Coordinates test activities with other TRS support teams

• Assists QA team with test cases and data set up

• Prioritizes and tracks issues to resolution

• Verifies and reports final test results

Database Administrator

• Performs database backup and restorations, as required

• Assists in identifying potential database issues/obstacles

• Monitors/Analyzes database servers logs for issues and provides feedback

• Reviews Performance Test Reports

GECS Lead • Monitor resource utilization while running batch processing /Loads

• Assist in identifying potential batch job issues/obstacles

Development Manager

• Resolves application issues as identified by the performance test team

• Assists in identifying and resolving code issues Infrastructure Engineer

• Assists the Contractor in installing testing tool software and associated hardware with appropriate integration

• Monitors the application under load

• Performs System Availability/Failover tests with the assistance of the

Contractor

• Provides support for performance test environments

• Monitors/captures required Server metrics using Solarwinds during test times

• Receives notification of performance test timings to alleviate network load concerns

TEAM Performance Testing Services

Role Description

• Reviews the Performance Test Report and helps identify server performance obstacles

• Identifies scalability issues and recommends appropriate resource utilizations

Information Security

• Receives notification of performance test timings to alleviate network load concerns

Release Team • Ensures environment is available to test performance

• Coordinates releases and deployments to support testing team

• Receives notification of performance testing timelines

iv. WSS myTRS Project Milestones

TRS expects the Contractor to meet the following project milestones for the WSS myTRS application. Submitted Responses should reflect consideration of these milestones, as well as the corresponding deliverables outlined in the table which follows. See section IV.

Evaluation Criteria, D. Technical Approach for additional details.

2020/11 - WSS myTRS project started 2021/04 - Performance Test Plan complete (Deliverables Group 1) 2021/05 - WSS myTRS application code is complete 2021/05 - Performance testing starts (Deliverables Groups 2 & 3) 2021/06 - WSS myTRS regression testing (1st round) is complete 2021/06 - Performance testing code remediation starts 2022/10 - WSS myTRS regression testing (final round) is complete 2022/11 - Performance (final) testing complete (Deliverables Groups 3 & 4 & 5) 2021/11 - User Acceptance Testing is complete 2022 Q1 - Final regression testing is complete 2022 Q3 - WSS myTRS Go Live

v. Deliverables

Group Deliverable Description

1 Performance Test Plan

• Must document the purpose, scope, and methodology of performance testing

• Must include:

o Business requirements o Metrics to be collected from servers o Test data requirements for each selected test case o Test cases for Performance Test Scenarios o Performance scripts parameterized with test data and service level requirements for configurations o Plans for executing each identified online and batch processing scenario o Plans for generating performance test result reports listing results by test case

TEAM Performance Testing Services o Demonstration that identified online and batch processing scenarios can be completed within the required processing time o Demonstration that performance of identified online and batch processing scenarios will not be affected by peak user load o Description of the bug reporting process o Plans for creating a final performance test report

1 Test Cases Must define the conditions to be tested and their assigned priorities. Prerequisites, detailed test procedures, and expected test results must also be included.

1 Test Scenario Workload Profiles

Must define a workload profile for each Test Scenario type

1 Test Scripts Must account for automated instances for selected test cases

2 Testing Progress Reports (ongoing)

Must provide regular testing progress reports based on the following metrics:

• # of test cases planned

• # and percentage of test cases executed

• # and percentage of test cases closed

• # of defects by status (severity and priority)

3 Analysis and Recommendations

Must analyze and recommend changes to increase performance and remove performance obstacles (must include run iterations and other supporting documents to demonstrate resolution of issues)

4 Performance Test Results Report (final)

Must include a summary of the final outcome of test execution for each testing type:

• # of test cases planned

• # and priority of open test cases

• #, severity, and priority of unresolved defects

The summary must include the following metrics:

• Average load time: The average response time for all requests (measures overall user experience)

• Peak response time: The longest response time (compares peak response and average load times)

• Wait time: Sometimes called average latency (measures the amount of time a request remains in queue before being processed)

• Requests per second: The number of requests handled per second

• Memory utilization: The amount of memory required to process a request

• CPU utilization: The amount of time required for the CPU to process requests

• Error rate: The ratio of errors to requests

TEAM Performance Testing Services

• Transactions passed/failed: Measures factors that cause requests to fail

• Concurrent users: The number of active users (also called load size)

• Throughput: Measures the amount of bandwidth that is used during performance testing (typically measured in kilobytes per second)

5 Knowledge Transfer

• Must provide a Knowledge Transfer Plan

• Must provide a TRS-specific Operations Manual

vi. Acceptance of Deliverables

Deliverable Acceptance Criteria Contractor shall provide TRS with a proposed written acceptance criteria document (PAC) for each specific deliverable at least fifteen (15) days prior to Contractor’s delivery of the deliverable. TRS shall provide written approval or rejection of the PAC within three (3) business days. If TRS rejects the PAC, Contractor shall have two (2) days to provide TRS with a revised written proposed acceptance criteria document (RPAC). TRS shall provide written approval or rejection of the RPAC within three (3) business days. If TRS rejects the RPAC, Contractor shall have a final two (2) days to provide TRS with a second written RPAC. TRS shall provide written approval or rejection of the second RPAC within three (3) business days. If TRS rejects the second RPAC, TRS may terminate the Contract with no penalty to TRS.

Deliverable Acceptance Contractor shall provide written notification for submission of each deliverable (Submitted or Revised Deliverable). TRS will have five (5) days to review a Submitted Deliverable against the agreed-upon Acceptance Criteria. If TRS rejects the Submitted Deliverable, Contractor shall have ten (10) days to provide TRS with a Revised Deliverable. If TRS rejects the Revised Deliverable, TRS may terminate the Contract with no penalty to TRS. If TRS accepts the Contractor’s Submitted or Revised Deliverable, TRS will provide Contractor with written acceptance.

vii. Reporting

The Contractor will provide a weekly status report that shows at a minimum:

1. Progress against each deliverable using agreed upon objective quantitative measurements

2. Issues

3. Risks

4. Activities planned next week

viii. Accommodations and Equipment

The Contractor will provide its own equipment, supplies, materials and/or employees. TRS will provide a virtual machine log in for the awarded Contractor’s staff after background checks have been completed.

The Contractor is not required to be physically onsite.

TEAM Performance Testing Services

ix. Invoicing Requirements Contractor may only invoice TRS for the work associated with a deliverable and receive payment for the work upon successful completion and written acceptance of the deliverable by TRS. Contractor shall attach a copy of the applicable written acceptance with each invoice. Contractor shall aggregate all accepted deliverables and submit them to TRS on one monthly invoice. Copies of the invoice must be provided to Account Payables and the Contract Sponsor. Invoices must be submitted to PayableTeam@trs.texas.gov or mailed to Teacher Retirement System of Texas, Attn:

Accounts Payable, 1000 Red River Street, Austin, Texas 78701.

x. Communication Plan Upon execution of the contract, TRS will set up a kickoff meeting with Contractor’s Primary Contact. At the kickoff meeting, TRS and the Contractor will review all the terms of the contract and discuss:

• How to perform all the tasks outlined in the SOW

• Deliverables of the contract

• Timelines for delivery of the Deliverables

• Required forms, such as insurance certificates

• Terms of the contract

• Invoicing requirements

• Performance monitoring

• Contract close-out process

During the kickoff meeting, TRS will assign a single point of contact for the Contractor.

xi. Contractor Staffing

TRS reserves the right to request that Contractor replace any Contractor employees whom TRS judges to be incompetent, careless, unsuitable, or otherwise objectionable, or whose continued use is deemed contrary to the best interests of the State and TRS. Upon receipt of a formal written request from TRS’ Primary Contact, Contractor shall propose to TRS a replacement with the same or similar expertise as the individual being replaced who shall perform the services required by the Contract, subject to the approval of TRS.

Contractor may not use a subcontractor on this contract without first receiving written approval from TRS regarding the proposed subcontractor. If Contractor is required to submit a HUB Subcontracting Plan (HSP) and the proposed subcontractor is not listed on the HSP, an amendment of the HSP will be required. Any work to be performed by such proposed subcontractor may not be initiated until the subcontractor has been added to the HSP.

As set forth in TRS’ Background Check Policy, a criminal background check (CBC) of the Contractor’s staff is required if at any time during the life of the Contract, the staff assigned to work on the Contract meets any or all the criteria listed below:

• Obtains a TRS user ID or password

• Has unlimited security badge access for any length of time

• Has restricted security badge access lasting more than 14 calendar days mailto:PayableTeam@trs.texas.gov

Further, a CBC may be required depending on the level of access to TRS resources and facilities that is required to perform the services. The CBC will be performed by TRS and the Contractor is required to reimburse TRS for the cost of the CBC within 30 days of receipt of notification that the background check is complete.

xii. Good Faith Negotiations

The Parties will attempt in good faith to resolve any claim or controversy arising out of or relating to the execution, interpretation, and performance of Services under the Contract through prompt negotiation between representatives of the Parties who have authority to settle the controversy. Contractor shall comply with all applicable federal and state statutes, rules, and regulations. The Contract shall be governed by, and all disputes between the parties resolved in accordance with, the laws of the State of Texas. Any actions or remedies pursued by either Party shall be pursued in the courts of Travis County, Texas.

IV. Evaluation Criteria

Responses and Best and Final Offers (BAFO) will be scored under best value standards that are in the best interest of TRS using the following general categories, for which weights are shown:

Criteria A. Minimum Requirements/Qualification

Weight Pass/Fail

B. Statement of Qualifications and Experience 15 points C. Respondent’s Key Personnel Qualifications and

Experience 35 points

D. Technical Approach 30 points E. Pricing 20 points Proposal Content and Format 0 or minus 5 points Total Oral Presentation/Interviews

100 points 20 points

Overall Total 120 points

TRS may request that top-scoring Respondents participate in oral presentations or interviews.

A. Minimum Requirements/Qualifications

Items in this section will be evaluate on a PASS/FAIL basis. Only those Responses meeting all of the minimum requirements will be declared responsive and proceed through the evaluation process.

Responses that do not meet the following minimum requirements will be rejected as non-responsive and disqualified:

• Must have experience in Agile software delivery methodology.

• Must have successfully completed a minimum of two (2) performance testing engagements where the Contractor was the Prime Contractor

• Must have completed performance testing for a WSS application for a customer base similar to the membership size of TRS (1.6 million)

• Must have been in business for a minimum of 3 years

• At a minimum, all candidates will be required to have at least two (2) years of experience with Agile testing, understand the practice of testing software bugs or performance issues within the context of an Agile workflow and have incorporated Agile into QA planning, estimation and metrics.

• Contractor’s proposed staff must be based and working from within the United States

• Contractor’s performance testing software must be hosted, or in cloud located in the

United States

B. Statement of Qualifications and Experience – Selection Criteria: 15 points

i. Contractor’s Company Qualifications

Company Organization

Respondents shall provide a concise description of their company organization including at a minimum: type of business organization (e.g., corporation, partnership, sole proprietorship, etc.); date established; state of incorporation; Texas authorization(s) to conduct business; location of company headquarters; and relevant branch offices. This description shall not exceed three (3) pages. Respondents should emphasize the unique characteristics of their company organization in relation to how they can best respond to this RFP.

Company Ability to Perform

Respondents shall provide a concise description of company resources, roles and approach to meeting TRS’ requirements for Performance Testing of its WSS application. Respondents shall provide a general explanation and/or organizational chart which specifies project leadership and reporting responsibilities and includes the proposed number of personnel by discipline to be dedicated to performing Services for TRS. The overall description should include Respondents’ plan for managing resources, as well as an escalation hierarchy and contingency plan for handling any issues that may arise. This description shall not exceed five (5) pages. Respondents should emphasize the unique characteristics of their company’s ability to provide the best team to meet TRS’ requirements.

Company Past Performance

Respondents shall provide a concise description of their company’s past performance as it relates to meeting TRS’ requirements for Performance Testing of its WSS application. This description shall not exceed six (6) pages. Respondents should emphasize the unique characteristics of their company past performance in meeting TRS’ requirements.

Respondents shall submit information on the three (3) most recently completed contracts or contracts currently in progress for similar work to that described in the SOW. Three (3) examples need to be provided by each listed subcontractor. The example contracts may include contracts with federal, state, and local governments, as well as commercial businesses. Additional consideration or weight may be given to project examples similar to this project in type and scale or level of effort. If applicable, Respondents shall provide a list of representative clients for comparable projects and demonstrate at least four (4) years of experience in the type of work described in the SOW. Respondents shall include the following information for each contract and subcontract:

1. Name of contracting activity

2. Contract number, if applicable

3. Contract title

4. Brief description of contract or subcontract

5. Total contract amount

6. Period of performance

7. Project or program manager and telephone number

8. List of major subcontractors, if applicable

9. Description of the services and how they were delivered

If Respondents have no relevant past performance history, Respondents must affirmatively state that they possess no relevant directly related or similar past performance. If a Respondent does not indicate whether past performance history exists, the Respondent’s Proposal may be considered ineligible for award.

Respondents shall list all work performed for TRS as prime Contractor or subcontractor during the last twenty-four (24) months, including contracts, purchase orders, and sole source and emergency procurements. For each specific project, Respondents shall list the TRS contact associated with the project and the TRS division and section in which the contact is or was employed.

TRS will review and take into consideration when evaluating past performance any information gained about a Respondent from the Texas Comptroller of Public Accounts (CPA) Vendor Performance Tracking System.

Company’s Client References

Respondents shall provide at least three (3) client references, including names, contact addresses, and phone numbers, for whom Respondents have performed similar services as described in this SOW. If the clients listed in the Company Past Performance section above are for similar work, Respondents may list the same client references in both the Company Past Performance and Company Client References sections of the Response under Tab 2. Respondents shall not include client reference information for clients who refuse to release information about Respondents’ previous contract work.

Note: If TRS is unable to contact references because the client reference information supplied by a Respondent is erroneous or incomplete, those references will be counted as “unsatisfactory” and may result in a lower score. If Respondents do not provide the required reference information, they must affirmatively state the reason.

C. Respondent’s Key Personnel Qualifications and Experience – Selection Criteria: 35 points

Respondents shall provide a list of company key personnel deemed necessary to fulfill the requirements of this SOW. For the personnel listed, Respondents shall describe specific, relevant experience(s), including contract numbers, project descriptions and dates, and project officers/team leaders. Respondents shall also provide detailed resumes or curriculum vitae for all key personnel designated to take responsibility for certain segments of the Work.

Respondents shall include similar information for any subcontractors to be engaged under the Contract.

Respondents shall provide a general explanation and/or chart which specifies project leadership and reporting responsibilities and indicates which Respondent personnel will interface with TRS management. Respondents may include personnel deemed necessary for TRS to evaluate the Response. Respondents must provide at least two (2) client references for each key personnel being proposed. The reference information must include names, contact addresses, and phone numbers for clients for whom key personnel have performed similar services as described in this SOW. Respondents shall not include client reference information for clients who refuse to release information about Respondents’ previous contract work.

TEAM Performance Testing Services

Note: If TRS is unable to contact references because the client reference information supplied by a Respondent is erroneous or incomplete, those references will be counted as “unsatisfactory” and may result in a lower score. If Respondents do not provide the required reference information, they must affirmatively state the reason.

D. Technical Approach – Selection Criteria: 30 points

Respondents shall provide a concise description of their technical approach to meeting TRS’ requirements for Performance Testing of its WSS application. This description shall not exceed fifteen (15) pages. Respondents shall include in their description the highlights of their technical proposal, including an overview of its plan and proposed schedule. Respondents should emphasize the unique characteristics of their technical proposal in meeting TRS’ requirements.

Respondents shall generally describe their technical approach and the proposed schedule or timing for performing the services described in this SOW. This description is not expected to exhaustively illustrate all Respondent capabilities, but, instead, should provide examples of areas where a Respondent’s abilities can be of best service to TRS.

Respondents shall provide the following in their description:

• A detailed description of the technical approach for completing tasks working within TRS’ Agile methodology approach

• A communication and escalation plan

• A schedule (timing) of the Work that demonstrates that they can meet the Project milestones described in Section III. Scope of Work, B. Program Overview, iv. TRS WSS myTRS Project Milestones and provide the Project deliverables described in Section III.

Scope of Work, B. Program Overview., v. Deliverables in accordance with the Project milestones

• A payment schedule that includes the amount of the deliverable payment and the month in which they will submit the invoice for the deliverable

• Quality Assurance/Quality Control procedures for performing the Work.

In describing the technical approach, Respondents must demonstrate that they are able to competently perform all required Services in accordance with this RFP. Respondents should specify in their technical approach what performance testing tool is proposed, whether it is cloud-based or on-premise, where the software is located, and recommended hardware requirements for an on-premise solution.

Respondents may propose alternative schedules (timing) for performance of the Work, but in doing so, must explain why the alternative schedule (timing) would be in the best interests of TRS. If Respondents propose a performance testing tool other than LoadRunner, they must provide an estimate of costs for the purpose of the software and/or on-going licensing. Such costs will not be assumed by TRS. If offered, Respondents must describe any optional features, products, or services that relate to the requested Services and provide line-item pricing for such on the Mandatory Price Sheet in Section V.

E. Pricing – Selection Criteria: 20 points

Respondents must propose their fully loaded, firm, and fixed price. Prices shall include all costs of performance, including materials, equipment, labor, communication, and planning to successfully complete all Services. TRS will consider all proposed prices as fully loaded rates, that is, inclusive of all expenses. No other prices may be charged to or paid by TRS. Pricing will be evaluated by the evaluation team based on fairness, reasonableness, and the best overall value for TRS’ interests.

F. Oral Presentations/Interviews (Optional) – Selection Criteria: 20 points

TRS may…