ESBD_File_220222_21-0000096_Addendum_2_Final_Redacted.pdf

Solicitation Name: TEAM Performance Testing Services Solicitation No.: 21-0000096

Addendum No.: 2

Teacher Retirement System of Texas (TRS)

December 10 2020

ADDENDUM No.: 2

TO

SOLICITATION NO.: 21-0000096

FOR

TEAM Performance Testing Services

DEADLINE FOR SUBMISSION OF PROPOSALS: December 21, 2020, @ 3:00 p.m., Central Time

LISTED BELOW ARE ANSWERS TO SUBMITTED QUESTIONS, CLARIFICATIONS/CHANGES TO THE

SOLICITATION

ISSUED BY:

Aundre Petty, Contract Specialist Date: 12/10/2020

NOTE: VENDORS ARE REQUESTED TO ACKNOWLEDGE RECEIPT OF THIS ADDENDUM BY SIGNING AND DATING BELOW AND INCLUDING WITH SUBMITTED RESPONSE. RESPONSE MAY BE CONSIDERED NON-

RESPONSIVE IF THIS COVER PAGE IS NOT SIGNED AND RETURNED.

RESPONDENT SIGNATURE DATE

12/10/2020

Solicitation Name: TEAMS Performance Testing Services

Addendum No.: 2

ADDENDUM No.: 2 **** Important Notice to Vendors ****

SOLICITATION NO.: 21-0000096

FOR

TEAM Performance Testing Services

This Addendum is issued to provide answers to submitted questions, clarifications/changes to the solicitation.

A) Questions and Answers

1) How many personnel resources (people headcount) does TRS expect to fill this "TEAM Performance Testing Serivces" RFQ with? One or more?

- TRS Response: Referring to how many Respondent resources are expected, the answer is “the number necessary for the Respondent to satisfy our requirements and produce quality deliverables”

2) How important is it for TRS to only use LoadRunner as the Performance Testing Tool for this RFQ response?

- TRS Response: TRS is open to other toolsets. The Respondent, who has expertise with performance testing, should propose the tool that would best serve TRS.

3) What is TRS's Operating System/Browsers/Devices supported and any SLAs as it pertains to this project and the WWS myTRS application (portal and batch)? The RFQ (pg 9) implies that the application is expected to work on most popular smart phone and tablets, and one would assume desktops as well.

- TRS Response: The application works within a browser-based responsive design and on the latest supported version of Chrome, IE, Safari, Firefox and Chromium/Edge browsers.

4) Is TRS providing access to LoadRunner or will we need to have this completely covered in the RFQ (assuming LoadRunner is the Performance Testing Tool used)? This is asked because on page 18 of the RFQ the 'Accommodations and Equipment' section states, "The Contractor will provide its own equipment, supplies, material and/or employees. TRS will provide a virtual machine log in for the awarded Contractor's staff..." This answer can greatly impact cost.

- TRS Response: TRS will provide access to our instance of LoadRunner should that be the tool proposed by the Respondent.

5) Will TRS secure an additional 5,000 LoadRunner licenses for "The Contractor" to use when conducting Stress Testing scenarios which call for 10,000 users? This is a non-question if the answer to the above question is "No". This is asked because on page 14 of the RFQ the 'Tools specification' section states, "TRS currently has a load testing software license from LoadRunner that allows TRS to simulate up to 5,000 virtual users." However, on the same page in the

Addendum No.: 2

'Performance Test Service Level Expectations' table it lists 10,000 users as the range needed for Stress test load.

- TRS Response: If LoadRunner is the tool proposed by the Respondent, TRS will secure an additional 5,000 licenses.

6) Which of the roles listed in the 'Key Personnel' table on pages 15-16 will be provided by TRS ('Project Manager', 'QA Manager', 'Database Administrator', 'GECS Lead', 'Development Manager' 'Infrastructure Engineer', 'Information Security', 'Release Team')?

- TRS Response: The roles listed in the question are all roles supplied by TRS. Respondent should propose the necessary personnel to fulfill the “myTRS Performance Test Contractor“ role in the 'Key Personnel' table on pages 15, to ensure quality deliverables within the timeframe provided.

a. Which will be provided by the contractor ('myTRS Performance Test Contractor', others)?

- TRS Response: See answer to question #6.

b. Are any of the roles vacant? If it is this can greatly impact scheduling, quality, scope creep, and delivery.

- TRS Response: All TRS roles listed in the 'Key Personnel' table on pages 15-16 are currently filled.

7) Will TRS secure an additional 5,000 LoadRunner licenses for "The Contractor" to use when conducting Stress Testing scenarios which call for 10,000 users? This is a non-question if the answer to the above question is "No". This is asked because on page 14 of the RFP the 'Tools specification' section states, "TRS currently has a load testing software license from LoadRunner that allows TRS to simulate up to 5,000 virtual users." However, on the same page in the 'Performance Test Service Level Expectations' table it lists 10,000 users as the range needed for Stress test load. Can we reduce the Performance Test Level Expectation to 5000 users

- TRS Response: See answer to question #5.

8) What are TRS's Operating System/Browsers/Devices supported?

- TRS Response: See answer to question #3.

9) What are the existing SLAs, in addition to the 'Performance Test Service Level Expectations' table on page 14, that exists for TRS (if any) that impact this proposal and the WWS myTRS application (portal and batch)?

- TRS Response: There are no additional SLAs.

Addendum No.: 2

10) What product type of LoadRunner is TRS using? Enterprise, Professional, Cloud, Community, or a combination of these?

- TRS Response: Professional

11) What version number of LoadRunner is TRS using?

- TRS Response: LoadRunner 2020

12) What type of LoadRunner license does TRS have? Perpetual, Term, Virtual User Days, SaaS, or something else?

- TRS Response: Perpetual - LoadRunner Web and Multimedia Protocol Bundle

13) What is the current protocol/Vuser type breakdown for the current Load Runner 5,000 virtual user licenses?

- TRS Response: Web http/html

14) What is defined by "large" for the 'Performance Test Service Level Expectations' table for SLE 3 (page 14), "Pages with large data volume and/or calculation intensive during batch execution (e.g. Benefit Estimate)"? What threshold is set for the data volume to be determined as "large"??

- TRS Response: Large would include multi-year (e.g. greater than 5 years) and multiple records per year. This would include the following High-Level Scenarios listed in the table on pages 12-13:

o 2 - View Member Account Home Page (Active Member), o 3 - View Member Account Home Page (Benefit Recipient), o 5 - Perform Benefit Estimate (Active Member-only), o 8 - View Member Document Center o 12 - Maintain Tax Withholding

15) Are online and batch scenarios with load tests expected to be executed on a schedule, ad-hoc, or both? (based on page 14, Note under the 'Performance Test Service Level Expectations' table)

- TRS Response: Both

16) What is the exact list of WSS application components used for the myTRS instance? Page 11 under Portal Load Test lists, "(e.g., database, hardware, network, etc.)". These extras (etc.) can drastically impact scope.

- TRS Response: The awarded Contractor will have access to the exact list of WSS components used for the WSS application.

17) Will 'Portal Load Test' (page 11) only include online and not any batch scenarios?

- TRS Response: Both

Addendum No.: 2

18) What scalability scenarios examples and expected thresholds limits are available (as it is referenced in last sentence for 'Portal Load Test' on page 14)?

- TRS Response: TRS wants to test the scalability of a low, medium and high load.

19) Can we assume that 3rd party services such as e-signature and identity proofing are not included in the scope of work? And, if so, is High Level Scenario 9 'Apply for Refund (e-Sign)' on page 13 no longer relevant?

- TRS Response: Per page 9 of the RFP: “Note: The myTRS portal also uses some third-party Web services for functions like identity proofing and e-Signature. These services are not included in the Scope of Work (SOW).” However, TRS does expect to test myTRS components of the scenario that do not require 3rd party testing and should be included in the Respondents scope.

20) Does TRS have Performance Center (web version of LoadRunner) or just LoadRunner? Assuming just LoadRunner.

- TRS Response: Performance Center is not part of the TRS LoadRunner License. TRS has just LoadRunner.

21) Are production environment details available to document information from?

- TRS Response: Yes

22) How close to a production like environment will be created for the performance testing environment for this RFP project or does this need to be determined?

- TRS Response: 95% matching the production environment.

23) On page 16 of the RFP in the 'WSS myTRS Project Milestones' dates list, is the date of '? 2021/11' correct for the 'User Acceptance Testing is complete' milestone? Asking because the milestones above (2022/11) and below (2022 Q1) are for the year 2022 instead of 2021. Also, it is not recommended to have UAT before the final performance testing is complete.

- TRS Response: Please see Section B of this Addendum for the revised timeline.

24) You note an existing LoadRunner installation. Can you comment on the version, the types of virtual users licensed, and whether this is Performance Center with one or more controller instances? Do you have existing LoadRunner expertise on staff for the maintenance/use of your LoadRunner infrastructure?

- TRS Response: Answered in questions #11, 12, 13, and 20.

Addendum No.: 2

25) Please help us to understand the rest of the monitoring infrastructure. Do you have App Dynamics, NewRelic, Dynatrace, Splunk, DataDog, or other deep diagnostic/Real User Monitor/log management solutions in-place today?

- TRS Response:

26) Do you have a third party CDN provider that will be integrated with the application in deployment? If not a third party CDN provider, are you planning on leveraging an in-house CDN deployed through the state of Texas network?

- TRS Response:

27) We are not a HUB certified vendor. Do we have to include a HUB certified subcontractor as part of the proposal to move forward with the RFP. Please let us know. Please note that In the minimum qualifications, there is no requirement on the HUB certification.

- TRS Response: No, you do not have to include a HUB Certified Subcontractor as part of your proposal, but all Respondents must complete and submit a HUB Subcontracting Plan (HSP) with their submitted proposal as required on pages 27 and 28 of 41 (Section VI.). Responses that do not include a completed HSP shall be rejected pursuant to Texas Gov’t Code §2161.252(b). You may seek assistance in completing the HSP by contacting the TRS HUB Coordinator, LaTresa Stroud, at latresa.stroud@trs.texas.gov or the Assistant HUB Coordinator, Aidet Cooper, at aidet.cooper@trs.texas.gov.

28) In regards to the 3 references, can we provide the reference content and explain the work we completed but keep the name and contact number if down selected? Some of our past performances will be willing to speak about our partnership but only if down selected.

- TRS Response: All Respondent reference contact information is due at the time of response submittal. If TRS is unable to contact references because the client reference information supplied by a Respondent is erroneous or incomplete, those references will be counted as “unsatisfactory” and may result in a lower score. If Respondents do not provide the required reference information, they must affirmatively state the reason.

29) Why has TRS chosen not to use a DIR cooperative contract?

- TRS Response: TRS has made the decision to consider all available options and the Request for Proposal process was the best way to compare and choose the best response to meet our needs.

30) Would TRS consider a proposal that leverages a DIR cooperative contract combining services and software?

- TRS Response: See answer to question #29.

This answer has been redacted due to the confidential nature of the information and the damage it could cause to TRS.

This answer has been redacted due to the confidential nature of the information and the damage it could cause to TRS.

2/10/2021-

2/10/2021-

TRSJD7

Highlight

TRSJD7

Highlight

TRSJD7

Highlight

TRSJD7

Highlight

Addendum No.: 2

31) What is the budget for this project?

- TRS Response: TRS does have a budget for the project, but will not share that information as part of the solicitation process.

32) Can TRS provide technical details about myTRS for software and team sizing purposes?

a. How many pages total?

- TRS Response: The Portal is a web-based application not a collection of fixed web pages. The application consists of 20 – 30 screens that allow members to view, update, store information.

b. How many update/store/delete pages?

- TRS Response: See answer to question #32a.

c. How many hosts?

- TRS Response: We operate virtual servers that can float as needed between physical servers. There is sufficient capacity for additional virtual servers if needed.

d. How many servers, and how much memory per server?

- TRS Response: See answer to question #32c. For each server, the memory varies from 8 GB to 32 GB as needed.

e. Where is the application hosted?

- TRS Response: In TRS data centers.

f. Does TRS prefer SaaS or on-premises performance software?

- TRS Response: TRS prefers software to be on-prem.

33) If the vendor-recommended performance software (for example Dynatrace) is available through a DIR cooperative contract, will TRS acquire it through a cooperative contract?

- TRS Response: Purchasing options for Respondent recommended performance software will be considered at the time of selection.

34) II. Description of Work - Are there any security restrictions on the application (i.e. HTTPS, SSL, Firewall, other)?

- TRS Response: All traffic is encrypted via TLS/HTTPS between all applications and servers. No firewall needs for this RFP.

Addendum No.: 2

35) II. Description of Work - Is there any difference in terms of capacity or architecture between the environment tested and Production environment?

- TRS Response: See answer to question #22.

36) II. Description of Work - What are the assigned environments for performance testing (Production, Pre-Production, QA, Dev, UAT, Others)?

- TRS Response: Testing will be limited to a Pre-Production environment only, specifically designated for Performance Testing.

37) II. Description of Work - Is the testing environment shared with other applications?

- TRS Response: No. See answer to question #36.

38) II. Description of Work - Are there any known performance application issues or any defect? If yes please elaborate and provide more details.

- TRS Response: No known performance issues have been identified thus far. However, the Web Self-service portal was originally architected to be down during nightly batch runs and is now required to be up and running during batch execution. In addition, the portal will now be retrieving data from multiple data sources. TRS needs to know the impact these changes will have on the performance of the portal.

39) II. Description of Work - Please mention what are the most critical business processes for your application.

- TRS Response: The application is a member self-service portal so the critical business functions include member account inquiry, demographic and profile updates, appointment scheduling, document viewing, benefit estimates, applying for refunds and tax calculation and adjustment.

40) II. Description of Work - Please provide the Scrum Teams composition (e.g. number of developers, number of Qas, etc.)

- TRS Response: The Scrum Team composition will be provided to the awarded Contractor.

41) II. Description of Work - Please provide a diagram of your current LoadRunner environment (generator, controllers).

- TRS Response: TRS has 3 servers with LoadRunner 2020 and Controller on each one of them.

42) May we see any test results that the development team may have run in any form, however informally?

- TRS Response: There has been no performance testing on the new WSS myTRS application.

Addendum No.: 2

43) Are any previous test scripts of any kind available from any non-performance functional testing?

- TRS Response: No.

44) Is TRS interested in equipping the system's development team with the capability to do spot-checks on upper load internally - so that future load testing would not have to be put out to bid?

- TRS Response: Refer to the Knowledge Transfer scope of the RFP.

45) Given the architecture, a reliable performance upper-bound estimation technique exists. Is TRS interested in an up-front estimate of a performance ceiling given the current design dependencies or is this intended to only be an "after-development" project to verify performance ceilings have been achieved?

- TRS Response: TRS prefers performance estimates of the WSS myTRS application after-development.

B) Solicitation Changes and Additions:

1) Change to Solicitation:

Section III. Scope of Work - B. iv. WSS myTRS Project Milestones On page 16 of 41 of the RFP, the following change is made to the WSS myTRS Project Milestones.

The year for these two milestones were incorrectly labeled as 2022 in the original RFP and have been corrected to 2021:

2021/10 - WSS myTRS regression testing (final round) is complete 2021/11 - Performance (final) testing complete (Deliverables Groups 3 & 4 & 5)