9078 - Additional Questions and Answers.pdf

FLORIDA DEPARTMENT OF TRANSPORTATION

Procurement Office

605 Suwannee Street, MS 20 Tallahassee, Florida 32399-0450

Phone: (850) 414-4381

ADDITIONAL QUESTIONS AND ANSWERS DATE: May 23, 2025

BID No.: DOT-RFP-25-9078-SJ

BID Title: Enhancing FDOT Standard Specifications and Construction Guidance with Large Language Models

OPENING DATE: Tuesday, June 3, 2025 @ 10:00 AM EST

There were inquiries that were received timely; however, the Department inadvertently left them off of the original Questions and Answers document. The additional inquiries and the Department’s responses are provided below.

Also, there was an error in Answer 9 within the original Questions and Answers document. The term for the completion of the project is one year. The Go-Live Date is anticipated to be April 2026.

Bidders/Proposers must acknowledge receipt of this Questions and Answers Document by completing and returning to the Procurement Office, by no later than the time and date of the bid/proposal opening. Failure to do so may subject the bidder/proposer to disqualification.

Sherill Johnson, Procurement Agent

Bidder/Proposer

Address

Submitted by (Signature)

Failure to file a protest within the time prescribed in Section 120.57(3), Florida Statutes, or failure to post the bond or other security required by law within the time allowed for filing a bond shall constitute a waiver of proceedings under Chapter 120, Florida Statutes.

ADDITIONAL QUESTIONS AND ANSWERS

Question 21 Could FDOT provide access to or details to the most current "FDOT AI policy" referenced in Task 1? Specifically, what are the key requirements concerning data privacy, security, model types, and ethical considerations for AI systems?

Answer 21 See Answer 11 (located within the original Questions and Answers Document attached to the advertisement for this solicitation within the Vendor Information Portal)

Question 22

Regarding Task 1, are there any pre-existing preferences or prohibitions concerning specific Commercial Off-the-Shelf (COTS) Large Language Models (LLMs) or AI platforms? Is there a preference between open-source and proprietary solutions, assuming they meet FDOT policy requirements?

Answer 22 Not limited, but for Task 1, the research team will need to provide the literature review and recommendations of LLM or AI platforms to FDOT. However, the chosen system needs to review by the Department OIT before developing.

Question 23 Could FDOT clarify the expected format and level of detail for the COTS tool comparison matrix required in Task 1? Are there specific evaluation criteria that should be prioritized?

Answer 23 Not limited. See Answer 3 (located within the original Questions and Answers Document)

Question 24

Could FDOT provide a definitive list and the current digital formats of all "FDOT Standard Specifications for Road and Bridge Construction, Standard Plans, Handbooks, and Guidebooks" intended to be used by the AI system (as mentioned in Tasks 2 and 3)? Will these be provided by FDOT, and if so, in what format(s)?

Answer 24

Most of them are PDFs and Word documents(includes graphics and tables), but they are not limited to scanned pages and images.

https://www.fdot.gov/programmanagement/implemented/specbooks/default.shtm https://www.fdot.gov/design/standardplans/default.shtm https://www.fdot.gov/roadway/fdm/default.shtm https://www.fdot.gov/materials/administration/resources/library/publications/materialsm anual/default.shtm

Question 25

How frequently are each of the source documents (Specifications, Plans, Handbooks, etc.) expected to be updated? We understand the update schedule for certain publications but would like confirmation for all source documents. What process is envisioned for updating the AI system's knowledge base to reflect these changes?

Answer 25

FDOT Specifications Office currently updates the Standard Specifications on an annual basis, typically in November. To ensure the AI system remains accurate and aligned with the most current requirements, the envisioned process includes syncing the AI knowledge base with each annual publication cycle. Additionally, interim updates or critical revisions—such as Design Bulletins or Handbook revisions—can be incorporated on a quarterly basis or as needed through a structured change management protocol coordinated with the Specifications Office.

https://www.fdot.gov/programmanagement/implemented/specbooks/default.shtm https://www.fdot.gov/design/standardplans/default.shtm https://www.fdot.gov/roadway/fdm/default.shtm https://www.fdot.gov/materials/administration/resources/library/publications/materialsmanual/default.shtm https://www.fdot.gov/materials/administration/resources/library/publications/materialsmanual/default.shtm

Question 26

Task 3 states the AI system should use "only FDOT Standard Specifications...and Design Guidance". Does this strictly preclude the use of the underlying LLM's general knowledge, or does it mean the AI's responses must be solely based on and verifiable against the provided FDOT documents?

Answer 26 Correct. We would like the AI system to provide information exclusively from the Department, not from open sources.

Question 27 Does FDOT have any existing data or AI frameworks it expects the vendor to build upon, or is the expectation that the vendor will develop everything from the ground up?

Answer 27 No, the expectation is the vendor will develop everything.

Question 28 Is the goal to integrate this effort into any other software like Bentely?

Answer 28 We are not certain at this time, but integration with other software platforms like Bentley is a possibility depending on future needs and system compatibility.

Question 29 Can we confirm that the goal of this scope is proof of concept?

Answer 29 Yes

Question 30 Regarding the AI evaluation methodology in Task 2, could FDOT provide more specific examples of the "real-world objectives" the methodology should align with?

Answer 30

An example of a real-world objective that the methodology should align with is software resilience with respect to the version of the standards it uses to answer user queries.

The software needs to have a degree of robustness sufficient to account for changes in specifications. The methodology should be one that is capable of easily incorporating changes in existing standards and the addition of new standards. In addition, the methodology should also be capable of handling different versions of the standards since the version of the standards that a project uses is dependent on the letting date of the project (i.e. a project with a letting date of January, 15, 2024 would use FY 2023-24 standards).

Question 31

Beyond the mentioned goals of robustness, efficiency, practicality, and accuracy, does FDOT have specific quantitative performance metrics or targets for the AI system evaluation (e.g., minimum accuracy percentage, maximum hallucination rate, maximum response time, citation accuracy)?

Answer 31

(This answer updates the answer provided in Answer 15e of the original Questions and Answers Document.)

Since this is a developing research project, it’s difficult to provide an exact estimate.

However, the questions can be evaluated through Tasks 4 and 5. For the proposal development, the department recommends designing the system to support up to 10,000 concurrent users for high-traffic applications. The AI system should be able to process 50,000 queries per hour at peak load during a 24-hour period. Anticipate Response Time should be : 30 seconds.

As this is the first project, the expectation serves as a goal, while the specific requirements will depend on the project or system itself. The Department reserves the right to accept the system based on its actual performance.

Task 1 “This task will focus on reviewing and documenting existing literature on the use of Artificial Intelligence to summarize content from multiple sources. Common sources of performance errors and best practices in testing of products will be documented.”

Task 4 “PI shall work with the PM and the FDOT specification Office to introduce the system to FDOT districts and conduct the survey with FDOT users through workshops.

The Principal Investigator shall summarize the survey results and provide an improvement for the system.”

The pass/fail thresholds will be determined by survey results or a literature review with the Department’s approval.

Question 32 For Task 3 testing, will FDOT provide a standard set of structured questions, or is the vendor expected to develop the test questions based on the evaluation methodology?

Answer 32 The Department will not provide the questions; however, as outlined in Task 4, the PI will develop an implementation plan and conduct a user survey as part of this study.

FDOT internal users will be responsible for asking questions and testing the system.

Question 33 Will the awarded vendor have access to FDOT technical staff or SMEs throughout the engagement for feedback and validation of AI system outputs?

Answer 33 Yes

Question 34 Will FDOT be providing historical project data or documentation for training or testing the AI systems, and who will retain ownership of the final AI tools and outputs?

Answer 34 Most of the documents and data should be publicly available. However, the Department will retain ownership of the research outcomes.

Question 35 For Task 4, could FDOT elaborate on the expected scope and scale of the initial implementation and user survey (e.g., target user groups, number of participants)? Will FDOT facilitate access to personnel for surveys and potential workshops?

Answer 35 It could be online or in person. We anticipate that each district will have 10 people join, with the maximum number of attendees being 100. FDOT will maintain the ability/right to record for future education. PM and PI will work together to facilitate the survey.

Question 36 Regarding Task 5, what are FDOT's expectations for the system transfer in terms of technical infrastructure, hosting environment (e.g., cloud-based, on-premise), required support documentation, and ongoing maintenance responsibilities post-transfer?

Answer 36 It depends on the final product. The Principal Investigator will need to provide a system transfer plan, subject to the Department's approval. Ongoing maintenance responsibilities after the transfer are not required.

Question 37 What are the anticipated duration, user base size, and key success criteria for the piloting phase mentioned in Task 5?

Answer 37 See Answer 31

Question 38 Are there any existing FDOT IT systems or platforms that the proposed AI system should be designed to integrate with?

Answer 38 We are not certain at this time, but integration with other software platforms like Bentley is a possibility depending on future needs and system compatibility.

Question 39 Is there a page limit for the Technical Proposal (Part I) outside of the 10-page limit for the Executive Summary?

Answer 39 No

Question 40 Task 4: Has the cloud infrastructure been identified for the AI system?

Answer 40 No

Question 41

Task 5: Are you open to the Principal Investigator hosting the infrastructure and solution in their environment (i.e. SaaS) where the FDOT Standard Specifications for Road and Bridge Construction, Standard Plans, Handbooks, and Guidebooks remain in FDOT’s environment?

Answer 41 At a minimum, the system must comply with the Department’s AI Policy. The Department will provide the necessary resources and documentation for the PI to follow.

Question 42 Does FDOT foresee a period where procurement of software needs to occur from the chosen vendor after Task 2 AI evaluation methodology?

Answer 42 With the approval of the Technical Research Committee (TRC), the vendor should proceed with system development.

Question 43 How do you foresee the AI system integrating with current FDOT systems and processes? (For example, SharePoint, ERP System, Static Documents?)

Answer 43 Integration will be a future task. This system will serve as the prototype LLM/AI system for the Department.

Question 44 Task 2: Are any of the FDOT Standard Specifications for Road and Bridge Construction, Standard Plans, Handbooks, and Guidebooks handwritten or non-digital?

Answer 44 In this project, we would like to focus on the digital version. Handwritten or non-digital versions will be considered in a future task.

Question 45 How many policy documents are in scope and how often do they get updated?

Answer 45

It has not been determined, in tasks 1 PI and Technical Research Committee will work together to define the scope of work. In this project, we will focus solely on the digital versions. Handwritten or non-digital documents will be considered in a future task.

However, at a minimum, the expected scope includes the Specifications, Standard

Plans, and Design Manual which are updated annually or as needed.

Question 46 Can you provide more details on the specific functionalities you expect from the AI system? (For example, Summarize and retrieve information, textual and/or image information, hot links, etc.)

Answer 46

The expected functionalities will be to retrieve and summarize information from the source material. The output given by the LLM may include images taken from the source material if said images are relevant to the user’s query, however, the LLM will not be expected to generate new images. In addition, when answering a query, the LLM should include references and hot links to the specific areas of the source material that it used to formulate a response to the query. Furthermore, within the excerpts of the source material that the LLM outputs when answering a query, any references to other areas of the source material or outside links should be done using hotlinks.

https://www.fdot.gov/programmanagement/implemented/specbooks/default.shtm https://www.fdot.gov/design/standardplans/default.shtm https://www.fdot.gov/roadway/fdm/default.shtm https://www.fdot.gov/materials/administration/resources/library/publications/materialsm anual/default.shtm

Question 47 Could you please share how both the program office and IT teams are collaborating in relation to this solicitation?

Answer 47 The Program Office will work with the Principal Investigator (PI) to focus on the system’s performance and accuracy. The IT team will be engaged when there are security concerns or other technical questions related to the system.

Question 48 Are software, infrastructure, licenses, and consumption costs included in the $100,000 budget?

Answer 48 Per Addendum 2, the budget is now $150,000.00 and, yes, all related costs must be included within the budget.

Question 49 Does FDOT have a target number of users are aiming to use the tool?

Answer 49 See Answer 31

Question 50

I am writing to respectfully request clarification regarding the FDOT AI policy referenced in the Scope of Services for solicitation DOT-RFP-25-9078-SJ (Developing FDOT Standard Specifications AI for Road and Bridge Construction). Specifically, Task 1 refers to becoming familiar with and complying with the FDOT AI policy, but I was unable to locate a formal or standalone document that outlines these requirements.

Could you please confirm if there is a formal FDOT AI policy document available for reference, or if there are internal guidelines we should adhere to when evaluating and developing AI tools for this project?

Answer 50 Please see the Questions and Answers added May 13, 2025, attached to the advertisement within the Florida Vendor Information Portal.

https://www.fdot.gov/programmanagement/implemented/specbooks/default.shtm https://www.fdot.gov/design/standardplans/default.shtm https://www.fdot.gov/roadway/fdm/default.shtm https://www.fdot.gov/materials/administration/resources/library/publications/materialsmanual/default.shtm https://www.fdot.gov/materials/administration/resources/library/publications/materialsmanual/default.shtm