FLHSMV-RFI-036-22 - QAs.pdf

FLHSMV-RFI-036-22

Questions and Answers

FLHSMV-RFI-036-22 Page 1 of 6

June 2, 2022

Note: All written questions are reproduced in the same format as submitted by the Vendor.

Question(s) received on 5/13/2022 from Christina Miller:

1. Question: Whether companies from Outside USA can apply for this? (like, from India or Canada)

Answer: This is not a competitive solicitation; the Department is seeking information from vendors as outlined in the Request for Information.

2. Question: Whether we need to come over there for meetings?

Answer: Please see response to question No. 1.

3. Question: Can we perform the tasks (related to RFP) outside USA? (like, from India or Canada)

Answer: Please see response to question No. 1.

4. Question: Can we submit the proposals via email?

Answer: Yes, please see Section 11.0 RESPONSE SUBMISSION INSTRUCTIONS, B. Delivery of RFI submittal.

Question(s) received on 5/16/2022 from Deltek

5. Question: Do you happen to know the estimated funding source or budget for this project? If so, has funding for an RFP been secured?

Answer: No, the Department has no budget information currently.

6. Question: Is the ORION system the incumbent system? Was it made from an award?

If it is the incumbent contract, would you be able to provide the contract number, vendor name, and term of the contract?

Answer: Yes, ORION is the incumbent system. This system was built in-house using a system integrator.

7. Question: I understand this is not a solicitation, however, hypothetically is there a timeline for an RFP to be released?

If not, what steps are expected to decide on releasing an RFP?

Answer: No, a timeline has not been determined yet.

Questions received on 5/20/2022 from iLab LLC

8. Question: Our understanding is the applications under test are across mobile devices, web-based applications. Are there any applications that are mainframe?

Answer: The majority are web-based applications, but the Department has a small mainframe footprint that would not be within scope of this testing effort.

FLHSMV-RFI-036-22 Page 2 of 6

9. Question: Is there a budget for a test data management tool or is there one already in place?

Answer: The Department currently has a Test Data Management toolset in place.

10. Question: What is peak usage of the largest application? What technology is the largest application built on from a performance testing standpoint?

Answer: The Department’s largest application receives 12,000 – 15,000 calls per minute. The technology used is Windows and .Net Questions received on 5/20/2022 from EM Key Solutions, Inc.

11.

Question: Does the Infrastructure performance testing include Network performance testing (such as Network speed, Throughput, Download/upload speed, Jitter, Delays, etc.) as well or just the servers monitoring and performance?

Answer: No, the Department is seeking information on testing for hardware configuration, hardware performance, software stack, etc. Additionally, the Department is seeking information on testing of server-network intermediary devices (such as load balancers).

12. Question: Is the agency infrastructure on cloud or physical or is there a mixed environment?

Answer: The Department’s infrastructure is on private cloud.

13. Question: Does the agency manage infrastructure as a code?

Answer: No, the Department does not manage infrastructure as a code.

14. Question: Is the agency amenable to using cloud or on-premise COTS automation tools or looking for custom solutions?

Answer: Yes, please refer to the RFI Section 10.0 - Requested Information.

Questions received on 5/20/2022 from SLED Florida

15. Question: How big is the environment that they want to use for the testing? What is the number of VMs or microservices? The approximate number of application servers and estimated of RAM per server?

Answer: The Department has four environments: development, test, stage, and production. The Department has approximately 20 VMs per environment with about 30 applications on each .NET cluster.

16. Question: Is the test environment a cloud environment, on-prem environment, or hybrid?

Answer: The test environment is on private cloud.

17. Question: What load testing tool do they have in place today if any?

Answer: The Department uses JMeter.

18. Question: What technologies/languages are their applications written in that they are planning on testing?

FLHSMV-RFI-036-22 Page 3 of 6

Answer: The Department’s applications are written in the following languages and frameworks:

Biztalk, COBOL, C#, JavaScript, MS SQL, Oracle SQL, .NET, T-SQL, Typescript, Uniface.

19. Question: Are they looking for a SaaS Platform or a platform that can be hosted on-prem?

Answer: The Department has not decided on a dedicated approach.

20. Question: What incident/defect management platform are they using today if any?

Answer: The Department uses Service Now.

Questions received on 5/16/2022 and 5/20/2022 from iT1 Source

21. Question: How is Performance Testing done currently? Which Tool being used?

Current Metrics captured?

Answer: The Department uses JMeter for performance testing. The tool used is CPU

RAM. The current captured metrics are the number of transactions.

22. Question: What is the frequency of Performance Testing? Is it once in 2 weeks or a month?

Answer: The frequency of performance testing is done as needed.

23. Question: Does the existing License of Microsoft Azure DevOps include Test Plan component?

Answer: Yes, the existing License of Microsoft Azure DevOps includes the Test Plan component.

24. Question: Who will fill in the Customer references for previous work with federal, state, local, or county Government?

Answer: Customer references for previous work must be provided by the customer who received the respondent’s services.

25. Question: Is there any restriction on (Application, Data and Infrastructure) Access considering the vendor would operate from outside of United States? Any Non-Citizen Clause applied?

Answer: Please see response to question No. 1.

26. Question: Can the vendor propose multiple Tools such as UiPath Test Automation for Application testing and other tools for Performance Testing?

Answer: Yes, please refer to RFI Section 10.0 - Requested Information.

27. Question: Do we need to submit any Authorization document from OEM for the tools proposed?

Answer: No, there is no need to submit any authorization document from OEM for the tools proposed, as part of the response to this RFI.

Additional Questions received on 5/20/2022 from iT1 Source:

FLHSMV-RFI-036-22 Page 4 of 6

28. Question: Will FLHSMV only be selecting one testing vendor solution? In other words, is it a mandatory requirement for the tool/solution to have BOTH UI functional testing, performance testing capabilities (load, volume, stress), and app security test capabilities? Or is FLHSMV considering a combination of multiple best-of-breed that complement one another?

Answer: Please see response to question No. 1.

29. Question: Is there an expectation for the tool/solution to measure UI related aspects such as accessibility and usability also?

Answer: There is not a dedicated toolset, however the Department has used JAWS in the past to accomplish this type of testing.

30. Question: How critical is each of the different types of testing discussed in the Project

Goals? Please provide a % breakdown of # of tests.

Answer: All testing types are critical for the Department to enhance the quality of

Department’s application as part of a continuous testing strategy, however, please see the list below in priority order:

1. Regression testing.

2. Feature testing.

3. Integration testing.

4. Unit testing.

5. User Interface testing.

6. Load testing.

7. Infrastructure performance testing.

8. Application security testing.

9. Capacity testing.

10. Stress testing.

11. Volume testing.

31. Question: Please provide a list of applications along with associated technologies.

(e.g. Web, 3rd party Cloud, Desktop, Mainframe, Citrix, etc.)

Answer: .NET applications

32. Question: What is the business criticality of the applications that are to be automated?

A % classification of High, Medium, Low would suffice.

Answer: The business criticality of the applications that are to be automated will be determined at a later date.

33. Question: What is the typical release schedule for these applications? If possible, please share the release calendar for the next 1 year.

Answer: The release schedule varies depending on the project.

34. Question: What is the automation roadmap/plan for these applications?

Answer: Please refer to Figure 1 in the RFI.

FLHSMV-RFI-036-22 Page 5 of 6

35. Question: Does FLHSMV expect business users and non-technical resources to create and execute automated test scripts?

Answer: The Department is open to utilizing business users and non-technical resources dependent upon training provided.

36. Question: What level of testing expertise do the business users and technical resources currently have in your teams?

Answer: The business and technical users have experience with manual style regression testing using the applications.

37. Question: Other than Azure DevOps, what agile planning and/or test management tools does FLHSMV have in its current landscape? (e.g. Jira, Xray, etc.)

What CI/CD tools does FLHSMV have in its current environment? (e.g.

Jenkins, Azure DevOps, etc.)

Answer: The Department currently has TFS 2013, which is being upgraded to Azure DevOps 2019.

38. Question: What is the size of your team (manual testers vs. test automation engineers)?

Answer: The Department currently has approximately 40 manual testers.

39. Question: How many platforms and browser combinations does FLHSMV envisage to have for their applications?

Answer: The Department is open to the number of platform/browser combinations that will successfully maintain the automation solution.

40. Question: What automation metrics are collected and reported as of today?

Answer: The automated metrics are: limited build automation successes or failures.

41. Question: What legacy automation tools does FLHSMV currently have? (i.e. how is

Visual Studio 2019, ADO, and Veracode used in testing today?)

a. If yes, how many such scripts exist?

b. When were the scripts developed?

c. How frequently are they currently being executed?

d. How much maintenance is required?

Answer: No legacy automation tools are used by the Department.

42. Question: Please confirm is FLHSMV is open to vendor resources being located at offshore locations (outside of the US)?

Answer: Please see response to question No. 1.

43. Question: Does FLHSMV have any metrics/benchmarks for SLAs once a defect is identified?

Answer: No, the Department has not developed any SLA metrics.

FLHSMV-RFI-036-22 Page 6 of 6

44. Question: In addition to procurement, installation, support and training, is FLHSMV also expecting the vendor to provide services for preparation, execution, and reporting of test automation?

Answer: Yes, the Department expects the vendor to provide services for preparation, execution, and reporting of test automation.

45. Question: Does FLHSMV have any projects being executed in a water-fall mode? Or all project being executed using Agile and DevOps methodology?

Answer: The Department is delivering application with an Agile methodology, also the Department is moving to a DevOps method and a continuous testing strategy as part of that.

46. Question: Does FLHSMV have a regression suite identified for automation? What %, if any, has already been automated?

Answer: No, the Department has no regression suite identified for automation. None of the current testing is automated.

47. Question: Please confirm if FLHSMV is open to explore cloud-based test automation tools. What is the preferred deployment method of the solution?

Answer: Yes, the Department is open to explore cloud-based test automation tools.

Please refer to Section 10.0 - Requested Information.

48. Question: What is the duration for which FLHSMV is requesting the tool/solution licenses?

Answer: The duration of the tool licenses has not been determined.