Q and A FRA EDIMS from Live RFI-1.pdf

Federal Railroad Administration (FRA) Enterprise Data Integration and Management Support (EDIMS) Services

Live RFI #1 – 15 October 2024 Questions and Answers

NOTE: Please note, the below Questions and Answers are for INITAL market research. FRA has not settled on a technical strategy, acquisition strategy or contract structure at this time. Continued market research will inform the composition of the requirement going forward.

High Level Objective One: Data Engineering Support

Question Answer

1. What exactly is the type/color of the Data FRA is looking to obtain from any 3rd Party Global Authoritative 3rd Party Data Source Provider? Business Financial and Risk Data Intelligence?

What types of Data do you expect to ingest?

The vast majority of data that FRA manages is internally produced, including data reported by railroads directly to the FRA. In the few cases when FRA ingests data from 3rd parties, the 3rd party has already cleaned and validated it. (The data is silver or gold.) The challenge for the successful contractor will be to blend the data with FRA data and organize the combined data for analysis/reporting.

2. Is the current RSIS Data Lake an on-prem/hybrid/cloud infrastructure?

Can you comment on the data types moving through the system? Is this meta data? Text documents?

Engineering data?

RSIS is currently in an on-prem SQL Server environment, while there is a separate data lake in the cloud (MS data lake gen II). FRA IT will grab data from RSIS and land it in the data lake. The successful contractor will be responsible for ETL to convert it from bronze to silver and from silver to gold in the data lake environment.

The data moving through the system is almost entirely structured data that has been ingested by a transactional SQL database. The exception at this point is a growing effort to load LIDAR images. FRA has not engaged in analysis of the images yet, simply making them available to researchers. At some point in the future, we may explore using AI for analysis.

3. Do you have intelligence built into the data workflows to support automated improvement of data quality, or awareness of quality, as you move through bonze, silver and gold layers?

Not yet. A key goal of this initiative is to build automated data QA/QC rules/processes. FRA has Collibra as a tool that can be used by the successful contractor, though the successful contractor may choose other tools/approaches. Initial implementation of Collibra may be underway in parallel with this contract as FRA just began implementation.

4. Will the RSIS Sandbox tie into the CAIOs Advanced Research Network or will that be an FRA operated environment?

The Sandbox will be logically an FRA operated environment to begin with. We will need to look into whether this will connect with any other network.

5. Is Databricks being used at FRA? FRA does have a Databricks license. There has been at most limited use thus far, but it is available. It is the Azure Databricks service version of Databricks.

6. Does FRA has any preferred Cloud environment? Has FRA aligned on a cloud provider for this architecture?

FRA and DOT have two primary cloud providers- Microsoft Azure and Amazon Web Services. FRA has primarily aligned itself with MS Azure but would listen to justification for using AWS, if compelling enough. The existing data lake is MS data lake gen II, so FRA’s presumption is that the successful contractor will help expand that data lake into an effective data lake house.

7. What type of data, is it metadata? The data is overwhelmingly structured data ingested by transactional SQL databases.

There are a small number of binary (LIDAR) image files. FRA expects to begin more significantly managing metadata in the data lake house (and through use of Collibra).

8. Is the data from BIS going to utilized for reporting purpose?

Yes, BIS is providing data for several reports already. Once BIS is expanded into a data lake house with a dedicated presentation (gold) layer, it will be utilized very extensively for reporting. For example, FRA’s public safety data website is currently staged in RSIS itself (on prem SQL server environment), but FRA wishes to transfer staging to the presentation (gold) layer in the data lake house.

9. Does each inbound file have the same format or are they all different based on the data coming in?

The files are generally structured SQL files. There are some LIDAR image files.

Moving forward, FRA expects there may be some PDF or CSV files.

10. The diagram shows 'ELT' between data sources and the Lakehouse. Can you clarify whether Transformation will occur between sources and the bronze layer?

The successful contractor may do some initial data quality control in RSIS (on prem SQL server environment) before the ELT process transfers the data. FRA does not expect to perform any transformation in the process indicated as ELT in the diagram, (It was so indicated to be open to possibilities and suggestions from the successful contractor, but FRA does not itself presume that there will be any transformation, simply transfer.)

11. Is data being transmitted from rail operators? How much data do you approximate?

Yes, we anticipate that all Part 225 reporting will be transmitted via electronic submittal in the future (accident and injury reports, grade crossing inventory forms, and such). The rail operators submit through a portal or file upload and the data is ingested into RSIS (currently in a n on prem SQL server environment but planned to soon be migrated to an Azure cloud environment.) Anywhere from a few hundred MB to 1GB per month

12. After the Gold layer, what platforms are used for reporting and analysis?

FRA has several platforms used for reporting and analysis including Power BI, Tableau, Socrata, SAS, R, and Python.

13. Understanding that there's a warehouse and lake in place, is FRA open to buying a data platform (like a Databricks that has the medallion layers built in)?

There is a lake in place and something that has incorrectly been referred to as a warehouse, but which is not a true data warehouse. In any event, yes, we are open to that option and and other options that provide similar capability if the successful contractor makes a justifiable case. FRA has Databricks Azure services available, but we are open to procuring a Databricks environment if that would provide an advantage over Databricks services in an Azure data lake.

14. Are any of the data sources raw data coming from sensor systems?

Not at this time. There are LIDAR images coming from automatic track inspection systems, but these are binary image files, not raw sensor data streams. FRA does not have IoT or real-time monitor data streams. In the future, we may ingest pre-processed “sensor” data from 3rd parties or from drones.

15. What is existing technology in place that must be supported? Data Lakehouse, ETL tool, BI tool?

FRA has a number of existing technologies in place but is counting on the vendor to assist in determining the best approach to be adopted across the Enterprise. FRA has a data lake (MS Azure data lake gen II) and generally uses MS Azure databases. FRA has Collibra for data cataloging/data governance.

16. Is the tech stack already set for this architecture? Which technologies?

Yes and No. FRA has an existing data lake \ (MS Azure data lake gen II, referred to as BIS) and presumes absent recommendations from the successful contractor that this data lake will be expanded functionally into a data lake house, but FRA has not committed to a specific product or technology to allow the Medallion architecture.

FRA wants recommendations from the successful contractor.

17. How many disparate systems do FRA has for this project?

FRA has several systems but will concentrate on just a few initially. Over the course of the contract, other systems will hopefully be added, as this is an Enterprise level project.

18. Where are the current data sets now? Disparate databases and/or vaults? Windows directories? Is some data being housed by contractors? Is data migration part of the scope?

Data are housed in the transactional information systems (SQL server or Azure) in which the data is ingested, and copies of these databases are stored in the data lake (BIS). No data is being housed by contractors. Data migration is planned to be handled by FRA IT separately rather than being in scope for this contract.

19. Are there external data feeds to be ingested in this data lake house? Will they have a different ingestion process compared to internal data?

At this time, no, and FRA plans to ingest externa data in batches in the near- and medium-term future, but there could be external data feeds to be ingested in the longer-term future based on agreements with those external data owners. Yes, there could be different processes for ingestion of external data versus internal data, but it is important to note that we expect that the successful contractor will uniformly use batch processing at this time.

20. Which ETL tools/Business Intelligence Govt have?

FRA expects that ETL will be performed using SQL (MS tools) and/or Databricks (Azure Databricks service). If advantageous, ETL could be performed using Python, FRA is open to suggestions for procuring additional ETL tools. For BI (in terms of data exploration and reporting), FRA uses Power BI, Tableau, Socrata, SAS, Python, R, and other tools.

21. Do you foresee leveraging a common data framework/format or data schemas? Thank you.

FRA is open to using a framework that enables analysis of data in a holistic and integrated manner rather than in silos including through adopting a common data/reference architecture. FRA recently procured Collibra to help execute on a data governance framework and consistently define all data fields and schemas.

22. Do all inbound files have the same schedule of when they send data or are they all different? Is the data coming in, new, updates?

Inbound files do not all have the same schedule. Some are close to real-time such as the inspection data and grade crossing inventory updates. Others are sent each evening and others are monthly. Data management updates can be performed by the successful contractor in batches, because FRA does not provide (or aspire to) real time data reporting.

23. Can you enumerate the data sources for this system?

This varies and will increase over time as more data is added. Initially, the successful contractor will be responsible for managing RSIS data. Over time, data from additional sources will be curated and joined/blended with RSIS data to create an expanded foundation for data reporting/analysis. For example, violations data from FRA’s “RCS” system might be second. There are tens of information systems that FRA uses that could potentially be sued as data sources, but in practice few will be curated for use in creating the gold layer, and at first it will just be RSIS.

24. For your meta data, do you have an approved corporate business glossary or common language that can address data integration?

FRA very recently procured Collibra to serve as our data cataloging/data governance platform. It will include a business glossary and a data catalog. FRA intends for the successful contractor to access and utilize Collibra to support integration and use of consistent definitions.

25. Is there differentiation as to what domain(s) of data get ingested to RSIS versus kept out of?

Yes, RSIS data is very specific and typically defined by regulation (49 CFR Part 225)

26. Are any of the data sources remotely located, what challenges are there moving remote data over distance to the data lake or data warehouses (latency, volume of data, data types, etc.)?

At this time, no. Data sources are generally internal FRA systems and FRA IT will transfer data and land it in the data lake for use by the successful contractor. There are files uploaded by our research lab in Colorado and some data from PHMSA (another DOT agency), but generally the successful contractor will be responsible for managing data in 2 locations: RSIS (on prem SQL server environment) and the data lake/data lake house. The successful contractor will not be responsible for moving data into these environments, just for managing the data in the environments. There are no meaningful latency issues because updates can be done in batches, and FRA aspires only to monthly reporting updates, though perhaps as frequently as daily for analytics. Definitely not real time.

27. Is there differentiation as to what domain(s) of data get ingested to RSIS versus kept out of?

Same as Question #25

28. Will there be any manual intervention within the clean layer where someone will need to fix/update and feed back into the silver to gold pipeline?

FRA does not anticipate manual intervention but understands that it may be needed in exceptional cases, though not as the norm. This will need to be taken into account and allowed for on those unusual occasions.

29. The top line of the diagram states a lakehouse, however, the next line shows a Data Lake, SoR, and Data Warehouse - which is more akin to a traditional two-tier data architecture, with multiple ETL processes in between. I want to confirm that FRA is looking to move from a two-tier data architecture to a Lakehouse Architecture, and want to understand how things like metadata capture and indexing of data will occur.

The equivocation in the diagram was intended to indicate some openness by FRA to recommendations from the successful contractor. FRA’s presumption is that a data lake house architecture incorporating FRA’s existing data lake may be the simplest solution but is not committed. FRA currently has a data lake but does not have a true data warehouse (even though we have a set of tables in RSIS that some people refer to as a data warehouse, in actuality it is not a real data warehouse). As such, FRA does not even have a traditional 2 tier architecture at this time. Metadata capture and indexing will generally occur in the source information systems, such as RSIS, and be landed in the data lake. FRA also procured Collibra to support denoting lineages and similar tasks.

30. Can you provide the projected data sizes for the Bronze, Silver, and Gold tiers?

Not at this time, but the data sizes will grow for each layer as this is applied across all of FRA To get to the (implied) point, the data sizes are relatively small by data management standards, not requiring any “big data” processing solutions. For example, SQL tools should be able to handle the data sizes, with no need for no-SQL big data solutions such as Hadoop. Databricks (Azure service version) is available to help with speed of processing as well.

31. What data types “structured/unstructured/semi-structured” are expected and any idea of sizes?

Almost all the data is structured (ingested by SQL/Azure data environments). Certain unstructured data is started to be collected (e.g., LIDAR images) but these are generally either from a specific project(e.g., research). In the future, FRA may begin collecting some unstructured or semi structured data (e.g., PDF files). Regardless, the total data sizes are small enough that big data solutions should not be required.

32. What are the current data volumes (both in terms of size and velocity), and how do you anticipate these volumes growing in the next few years?

In terms of typical data management standards, the volume is small, and the velocity is low. Reporting is updated monthly, and FRA aspires to at most daily updates for analytics. Excluding LIDAR images, total data is in the gigabytes range, not terabytes.

33. Are there existing IdM controls (either FRA or DoT) to leverage for internal/external information sharing controls? Or will the contractor team recommend as part of the solution?

There are not as of now, and the successful contractor will make recommendations as part of this contract.

34. Is FRA primarily utilizing batch or streaming processes for ETL workflows, and what factors led to that decision?

FRA uses Batch processes. FRA does not have a need for true real-time reporting or analysis, so batch processes are fine.

35. What are the current ETL completion times for the Bronze, Silver, and Gold processes?

Within the same layer, there could be different completion times, based on the multiple projects and need. Using just RSIS as the example, there are several different completion times for inspection data, accident/injury data, and grade crossing inventory data with some real-time and daily, and others on a monthly completion.

Per this contract, the vendor will become familiar with the entire inventory of known data sets and processes being run.

36. Is there a specific sequence in which these files should be ordered and processed?

Yes, there can be specific sequences in which files need to be ordered and processed.

Data owners would share that information.

37. Are you going to be using SWES for this work?

The contract vehicle for the FRA EDIMS requirement will be dependent upon many factors and is unknown at this time.

38. Are the ETL processes part of the data catalog/governance model? How will the processes be measured as successful?

There should be an awareness of the ETL processes as the model is developed. The vendor will help to determine the measurement for success

39. Beyond what was discussed today, is there a published data strategy that we should be aware of?

FRA CDO has drafted a data strategy which is currently in review. Once finalized, it will be shared with the successful contractor.

40. Has FRA already aligned on a specific solutions and cloud provider for the future architecture?

FRA is able to use Microsoft Azure and AWS, as both are DOT approved Cloud Service Providers. FRA will keep an open mind to all options but is using MS Azure in several projects already and would prefer, for continuity and integration, to stay with Azure.

41. In terms of innovation does FRA has an initiative to make the data infrastructure ready for AI use?

DOT and FRA are planning piloting certain AI initiatives while developing general guidelines for adoption of AI. The proposed sandbox (shown in the diagram) will be a key testing space for experimenting further with AI.

42. I do not see Data Quality/Data Governance?

See High Level Objectives two and five below.

43. Are there business rules that applies to the cleanup and does each file have their own set of business rules?

There are legacy rules that apply but EDIMS will set up new standards and expectations across all data files

44. What are the Data Sources besides

BIS?

The data sources are varied. With RSIS there are Inspection, Accident/Injury, and Grade Crossing Inventory that come from data collections from inspectors and railroad users, respectively. Each FRA program office has their own data sets. RSIS is on-prem and the others are on cloud servers.

45. Is there a recommended technology stack for this part of the solution?

FRA is open to your recommendation(s) in regard to the data solution, though the presumption is that FRA will leverage the existing MS-based infrastructure, including Azure SQL databases and Azure data lake gen II.

46. Can you elaborate on the movement of data between environments being managed by a separate contract?

FRA will use its contractor team to provide the ELT to grab data from source information systems (transactional databases) and land it in the data lake where the successful contractor will engage in data management with it. The contractor will also perform some basic cleaning and validation in RSIS before it is transferred to the data lake.

47. How come data movement between environments is managed via multiple contracts?

Movement of data from source systems to the data lake/data management environment will be handled by FRA IT and its O&M contractor. Data management will be handled by RDI (and its EDIMS team). Previously, work was divided by system. IT owned certain systems and RDI owned other systems. Moving forward, operations and maintenance will be handled by IT and data management by RDI.

Transferal of data from one system to another falls into operations. Cleaning, validating, transformation, and preparation of data for reporting and analytics falls into data management.

48. Have you considered how to visualize, and distribute (download) lake using software with a map user interface?

Yes, this is already done by the FRA GIS team.

49. Is there an existing Data Management Framework today or is the proposal to build this from scratch?

There is a legacy data management framework, but this contract will build a modern data management framework to build upon and supersede the previous framework and update it for a modern architecture and data analytics pipeline. There are some relevant existing elements that will need to be used.

50. Why is the data management framework under the data engineering high level objective? Often management frameworks include factors relating to upskilling individuals, business process and policy development, training, and engagements with end users to develop use cases for analytics -which is work that is typically not considered data engineering. Should these other factors be included, or

The data engineering high level objective sets the stage for the entire data management framework and the other high-level objectives. Many of the factors mentioned are covered by the other HLOs and the vendor will have a hand in developing each of them.

should they be ignored for this objective?

51. What is current policy to enforce data tagging/identification and overall lifecycle?

There currently is not a formal policy regarding tagging and needs to be developed and adopted as part of this effort under the guidance of DAPR. The data governance platform that FRA has recently acquired will be key in this effort

52. For the ETL process will the systems be on cloud only or is there a mix of on prem and cloud based systems in use?

FRA has a hybrid system currently, with a mix of (a single) on-prem system and the rest cloud based systems. All systems will be cloud based in the near future as FRA transitions the last remaining on-prem to cloud over the next year. The ETL processes will be constructed in a cloud-based system.

53. Something to consider (I'm struggling with separating the tech implementation contract from the data engineering support) - as you're building the new tech, your engineers may be required to write contracts, construct notebooks, etc. to move the data from source and then through each layer - data engineering is a large part of developing those processes and data flows - have you thought through the separation of engineering from the tech implementation?

FRA, like many other DOT agencies, has separated IT functions from data management functions. EDIMS will require teamwork between the data engineering and technical folks to implement the data engineering recommendations. To be precise, the EDIMS contract is not for building infrastructure. The EDIMS contract is for data management in infrastructure that is already built or will be built under the guidance of IT.

54. What is FRA currently using for 3rd Party Data Sourcing?

There are a variety of approaches being used to pull and be pushed data from internal and external sources, but each has been developed based on specific need and not with consistency and enterprise view in mind.

55. Have you considered cost around technology stack selection? Do you plan to integrate Generative AI in the roadmap down the line?

The FRA will be considering cost for any proposed solutions. FRA will adopt the best possible solution while keeping cost in mind. AI will be a key feature in the future and FRA is open to its use in accordance with DOT policy and procedures.

56. Are you committed to Medallion, or would you consider Lambda Architecture for its real-time and batch processing benefits, especially for

FRA currently does not process any streaming data. In context of the current state of art, data volumes are also relatively small. As such, FRA considers the Medallion architecture or something similar to likely be a more appropriate solution. While Lambda Architecture has benefits for big data batch processing there are also handling both live train data and historical analysis?

complexities that we wish to avoid, and it does not offer the same multistep approach that would utilize resources that FRA already has in place.

57. Are you open to moving off a licensed architecture to a GOTS / OSS model?

FRA has stated it is open to other options as long as they are comparable to Medallion.

Government-Off-the-Shelf and Open-Source Software would need to be comparable.

58. What specific data collection methods does FRA envision for this project? Are there preferred technologies or tools for data collection that you would like us to incorporate?

See High Level Objective #4 for the Data Collection requirements. FRA is seeking recommendations, in regard to preferred technologies and tools for data collection.

59. We are seeing a lot of value in using AWS and AWS Services paired with Open Source Software for our DoD massive scale data projects.

Would be great to understand volume of ingest a day, hot v. cold storage volumes, and number of data feeds?

FRA can use AWS or MS Azure, and FRA is using several Azure tools already. Data volumes are relatively small, and velocity is low. The number of data feeds is small.

Almost all data is structured. These sorts of structural challenges are not present for this project. Rather, the challenges primarily revolve around poor data quality and the need for robust governance and accuracy.

60. Would you consider using open source applications or tools with GovCloud?

As stated in other answers, FRA will consider all options but is already using MS Azure for its data lake and ETL/ELT tools.

61. Are there existing geospatial tools to integrate with the new data architecture? or will the contractor need to recommend new geospatial capabilities?

FRA would expect to continue to use Esri geospatial software, through our US DOT enterprise licensing agreement

62. Not seeing any mention of error handling. Assuming we would need that?

Yes, error handling is expected and would be covered by High Level Objective #2

63. What technologies or patterns (e.g., Debezium, Kafka) are currently used or preferred for managing change

Thank you for bringing up management of change data capture. Logs are very important, and FRA would welcome recommendations in this area. This would be part of High Level Objective #2.

data capture, and how are these logs utilized in downstream analytics?

64. Could you clarify the role of the separate contract managing the movement of data environments? How will this interface with the work FRA seeks to have done under this contract?

FRA’s IT contractor will handle the movement between environments and work together with the EDIMS team and the data owners to make this a smooth process.

Basically, IT and their contractor will land the data in the data lake, and then the successful contractor will manage and prepare it for use in reporting and analytics.

FRA will facilitate close coordination between teams.

65. Is engineering data of special concern? Is there CAD data or other design data that is a major data set that needs to be managed?

The term data engineering is referring in our usage to the process of putting together an overall data management approach and not data from engineering projects per se.

There is no “engineering data” such as CAD data.

66. Databricks and Synapse are only needed for distributed processing of very large data sets. I don’t see big or very big data HLO. Are you able to clarify why such tech is being considered?

FRA wants to consider architectures and processes that may provide flexibility into the future for enterprise level, and FRA has a license for Azure Databrick services that can easily be leveraged if helpful. Thus, FRA is open to recommendations, though FRA expects recommendations of tools to be such that they can support potential future uses

67. What is the sensitivity of the data?

Any classified info?

The sensitivity of the data is mostly Low and some Moderate. Within the next year, the data lake will contain PII and market sensitive data (some of which requires an NDA to access). Eventually some classified data may be added.

68. Does EDIMS need to conform to any larger DOT constraint -considering standards, policy, regulations, and enterprise architecture and controls?

Yes, EDIMS will need to conform to any and all DOT and US Government standards, policies and regulations regarding architecture and controls

69. To design the appropriate data architecture & data configuration in support of AI/ML & data pipeline(s) , do you have a common corporate language?

If this is in context of having a glossary, please see answer to #24

70. Has FRA initiated Data Governance framework with data

Please see High Level Objective #5 standard/quality, ownership policies and procedures?

71. Are there any other product licenses that have been bought and not used by FRA? Maybe we can bring together a bunch of them as an interim solution before moving to a higher quality architecture. Would be great to understand all of the tools that are already paid for. We can then architect a highly reliable solution that works across clouds and OSS / licensed platforms. Putting data into the solution that best fits it.

FRA would like to have the vendor make suggestions as to the architecture, regardless of the tools or licenses that have not been used by FRA. The one license that has not yet been leveraged is for Azure Databricks services. FRA is using and expects to expand use of several products in this project, including the full MS suite (MS Azure, MS data lake gen II) and Collibra.

72. Roughly how much of the data is documents or unstructured?

Almost all data is structured, ingested through SQL transactional databases. There are some LIDAR and image files and more semi-structured or unstructured data may be collected in the future.

73. Do you intend to use PySpark or Scala Spark for ETL/ELT?

Given the low volume of data, FRA expects that SQL may be the simplest option.

FRA does have an Azure Databricks services license, so Spark could be implemented if the contractor recommends it.

74. Has FRA discussed an enterprise data governance group for overseeing reference/master data? For example, to ensure data can be analyzed across different applications, many organizations standardize the schema for core data (what is a railroad? what are the coordinates along railroad tracks?). If so, has this data governance group already been stood up or would FRA consider creating such a group in parallel with its enterprise data platform effort?

Yes, FRA has stood up a data governance group, Data Architecture and Review Planning (DAPR) board which is filling that need of overseeing the governance plan.

FRA recently procured Collibra to support governance.

75. Similar to the data lake house architecture vendor question, would the same be true for data catalogs?

Collibra was mentioned but there are other offerings in the space.

FRA has completed procurement of Collibra after due diligence in market research and is committed to it as the data catalog solution for the near term.

76. How is the data being secured today at rest and in transit? Is FRA using a Security Information and Event Management (SIEM) tool today? How does this teamwork with the Security Operations Center

(SOC)?

FRA uses DOT approved processes to secure the data in transit and at rest for each of its data sets, but there could be more than one since we are talking about several dozen data sets. Each works with the FRA security team, where all data is scanned weekly and they in turn work with the FAA SOC

77. How do you intend to develop scalable architecture?

FRA is open to input from the successful contractor on developing of a scalable architecture

78. Are there existing metrics or standards of data quality that FRA wants to meet? Or do those baseline metrics need to be established?

Yes, there are existing metrics or standards of data quality to be met, but they can be enhanced and improved across the Enterprise. See High Level Objective # 2 for more on Data Quality.

79. Are there formally assigned roles and responsibilities for data owners/stewards to support data quality efforts? Is there formal policy with FRA?

Yes, there are formally assigned roles and responsibilities for all data owners/stewards in support of data quality efforts. The successful vendor will help implement the policy and maintain the artifacts in coordination with the governance group (DAPR).

See High Level Objective #2 for more on Data Quality.

80. What tool/solution are you using today for log collection and aggregation?

Currently, FRA is using tools internal to SQL Server and Azure (change data capture logs). FRA is open to recommendations for additional tools.

81. How is the data being secured today at rest and in transit? Is FRA using a Security Information and Event Management (SIEM) tool today? How does this teamwork with the Security Operations Center

(SOC)?

This is the same as Question# 76

82. How do you intend to develop scalable architecture?

This is the same as Question #77

83. Are there existing metrics or standards of data quality that FRA wants to meet? Or do those baseline metrics need to be established?

This is the same as Question #78

84. Are there formally assigned roles and responsibilities for data owners/stewards to support data quality efforts? Is there formal policy with FRA?

This is the same as Question #79

85. What tool/solution are you using today for log collection and aggregation?

This is the same as Question #80

High Level Objective Two: Data Quality

Question Answer

1. Do you have identified business rules to measure data quality across 6 dimensions? Do you also plan to have the ability to capture and compare snapshots of aggregated data?

We do not have business rules across six dimensions as such, but the successful vendor can help implement this. Capturing and comparing aggregated data, versioned data, could be considered a goal.

2. Are architecture/solution recommendations requested for Data Quality tools as well? Or is this objective specifically concerned with implementation?

Yes, your recommendations are requested as they pertain to data quality tools. Both hands-on and further ways to improve data quality are sought.

3. What data deficiencies has FRA seen thus far with respect to quality?

What are considered the hallmarks of high-quality data in FRA's context?

FRA seeks continued improvement in all aspects of data quality including some of the manual processes that are used to correct data quality issues. FRA data must be received on-time, processed in the time allowed and made available in formats that can be used to meet FRA needs.

4. Roughly - how many columns across FRA would be important to monitor in terms of data quality?

Roughly, what percentage of these columns currently have rules?

All data at FRA has to comply with rules but these may not be consistent from one data set to another, and across the agency and you would be responsible for monitoring the data quality of them all.

5. Is there still data that isn't available yet and needs to be collected? Also, how important is real-time data for your analytics needs?

Yes, FRA is constantly adding to the data it uses. A rulemaking can direct that new data is collected or a project may require that data be collected, such as long train data.

Real-time data is not required. Daily updates for analytics are sufficient.

6. Do you want data governance to be integrated into EDIMS or should they be separate procurements?

High Level Objective #5 deals with data governance. This is a part of EDIMS and not a separate procurement. FRA is implementing data governance along multiple dimensions, and the EDIMS contract fits into these general efforts. The successful contractor will coordinate and support FRA’s data governance officials but will not lead data governance for the agency.

7. What reporting tools we are planning to use. Power BI or MSTR?

FRA currently utilizes Power BI, Tableau, and Socrata, as well as some reports created in SAS. FRA is open to all DOT-approved reporting tools.

8. What vendor products/tools are used today for data quality maintenance that you will continue to use and integrate with in the target platform?

FRA recently procured Collibra, which has some data quality features. There are some legacy rules developed for RSIS which can be leveraged. The successful vendor will be expected to implement new data QA/QC tools/practices.

9. Do you need manual or programmatic correction?

FRA would prefer programmatic correction whenever possible but manual correction should be supported whenever necessary.

10. Would the agency be interested in a vendor with data aggregation and data quality experience for research for other agencies such as Veteran Affairs, HHS?

Yes, previous experience with data quality is valuable.

11. Do you anticipate streaming data from railway providers?

No.

12. We will have to create business rules around each dimension?

Business rules will be needed for many dimensions, such as checking that submissions use valid codes for that dimension.

13. Are you planning to integrate data quality tools into enterprise systems in line with the FRA's 2025 cloud migration goal?

Yes, the successful vendor will help shape which tools and how they are applied, in line with FRA’s future goals.

14. One more on data quality: Does FRA have time dependent processes, as it relates to data quality? For example, if FRA receives a data submission that needs to be fixed by the submitter, are there any systems/process at FRA that require the data to be fixed before systems/process can begin.

Yes. For example, per regulation, FRA receives certain safety data that is time dependent, then will process and post this data for public consumption. And if said data needs to be fixed or modified, FRA does have systems and processes that identify the data to be fixed, sends it back, and will not process it until it is fixed and returned.

Again, the goal is data management at an Enterprise level, so there will be several processes to keep track of.

High Level Objective Three: Data Information Architecture

1. Is FRA looking to capture lineage from on-prem systems into the medallion architecture as part of creating supporting documentation to support data governance?

Yes. FRA recently procured Collibra, which supports lineage documentation.

2.How do plan to define ownership across many vendors for these HLOs?

FRA has stated their preference for one vendor across all of the EDIMS High Level Objectives.

3. Is there an intent to eliminate data silos on legacy applications?

Yes, elimination of data silos would be one goal in a new architecture. By landing data from all systems in the data lake, that will allow the EDIMS contractor to curate, join or blend, and process data that provides integrated, holistic analytics and reporting.

4. What kind of data collection method you are looking for? Are we supposed to make some web form/GUI form to collect data?

FRA seeks expert guidance from the successful vendor on the information content and architecture for collection efforts. FRA IT will develop the infrastructure of the collection tools to meet the specifications of the EDIMS contractor. For example, if FRA needs to publish a new form to collect data from railroads, the contractor will specify what data fields need to be collected, the relationships between fields, valid codes for each field, etc. (i.e., the data model). FRA IT will make the web form or other collection instrument to those specifications, not the EDIMS contractor

5. Is there a backlog list of any bugs/gaps with the existing process that should be included/considered with this?

There may be a backlog of bugs but the successful vendor will be determining the logical and physical data models to be used across the FRA Enterprise.

6. What is the anticipated end user count for reporting at an enterprise level? Also, what is the YoY current system growth?

The end user count could be the entire FRA, approximately 1000 users, plus any external users, which goes into several thousand users. Usage will increase year over year as this is applied to the entire Enterprise. We can assume the Base Year will be a percentage of the whole and growing each year.

7. For future needs of FRA, when the new Medallion architecture is in place, do you plan to have Governance implemented at Row, Column Level

Data Governance is covered in High Level Objective # 5. The level of governance to be implemented will be at the row, column, and dataset levels. FRA complies with all requirements for sensitive information and regulatory compliance per FRA/DOT and US Govt.

and dataset level? Any sensitive information / regulatory compliances to handle here?

8. Does FRA currently have an Enterprise Data Model and practice governance via an Enterprise Data Management Office, to which EDIMS must conform? Is there any linkage to a DOT enterprise architecture and associated governance, such as via an FRA segment of enterprise architecture for DOT?

FRA DAPR functions as the enterprise data management team. There is not a formal enterprise data model yet, but the EDIMS contractor will help establish one including by leveraging and coordinating with any related activities.

9. Is FRA looking for any Data Web Apps for future needs?

FRA will consider all options.

10. I assume different departments (stakeholders) will have different requirements...both to the type of data they need and how they will use/consume that data. What can you tell us today about different types of end user requirements?

The assumption is correct, the different departments will have different requirements.

One group may want to utilize safety data while another uses research data or grant management data, and yet another uses legal data. Cross-functional data analysis is also done, with the examples of safety analysis of grant applications and legal analysis of safety data. The office running this contract (RDI-14) will gather requirements and communicate them to the successful contractor.

11. What are the technical challenges in extracting data from current legacy applications? (e.g. closed systems, older tech stack, etc)

The legacy source systems are all SQL-based (SQL server and Azure), so challenges should not be overwhelming.

12. Can you give us examples of the engineering data that will be used?

CAD data? CAE data?

No engineering data is currently managed. Mentions of engineering in the RFI were references to data engineering practices, not to engineering data.

13. What are the different data types that should be accounted for in the data models - structured, unstructured, geospatial, time series, and different media types?

Almost all source data is structured (ingested by SQL transactional databases). There are some LIDAR image files. FRA does make significant use of geospatial data.

High Level Objective Four: Data Collection Design

1. Where would FRA be collecting data from? Other government agencies, local/state government?

FRA primarily collects data from railroads and from inspectors. FRA gets some data from other government agencies, mainly PHMSA and state DOTs.

2. Could this objective also include business and technical “metadata” capture for crafting data products and datasets for end users?

Yes, this objective could include this

3. Are you looking to incorporate Geospatial Data into data collection?

Yes, FRA already has a large amount of geospatial data and will continue to collect this.

4. Would you collect weather data from other agencies, like NOAA?

FRA does not see a need for weather data currently but would consider it if there is a motivating reason.

5. And a quick follow-up, would FRA typically take actions to pull the data from external sources, or expect to receive the data as a push from such sources?

FRA currently pulls and is pushed data from outside sources, depending on the relationship with that external source.

6. Is the plan to use an existing ETL solution or a modular/custom python frameworks for data collections from different sources?

Existing ETL solutions will be leveraged by the successful contractor. These are generally solutions for data processing of RSIS safety data. Data collection mostly occurs through submission via web forms or file upload. FRA does use some APIs for ingesting data.

7. Overlapping comment between HLO3 and HLO4 - How is FRA looking to relate multiple disparate data models from different datasets that are created or collected? Has there been any though given to using things like ontologies or graph data structures to relate data models?

FRA is considering and open to developing and utilizing ontologies/taxonomies. FRA does not currently plan to employ graph data structures but consider it if recommended.

8. Is there any expectation that collected data will continue to change

Yes, certain data can change as it is the authoritative data used by FRA. For example, railroads have 5 years to update incident reports if additional information is at its source and may need to be kept current with its source? Is there a need that the lifecycle of data outside of the EDIMS environment be taken into consideration, or does FRA plan for and accept the possibility of "drift"?

discovered in an investigation or circumstances change, such as an injury becoming a fatality. The data lifecycle will be taken into consideration as data can change outside EDIMS and those updates will be needed.

9. Should scalability be addressed here or is possible addition of new data sources addressed elsewhere?

Yes, scalability should be addressed here with the Data collection objective.

10. In data collection, is there any consideration of Civil Right and Civil Liberties - being mindful of creating data for public consumption especially if there is a possibility of use of AI?

This was not addressed in this High Level Objective but it will be a consideration.

11. Has a decision been made to integrate data from external sources, such as third-party applications or government databases, into a new data warehouse, or integrate the data for real-time viewing without duplicating it, while ensuring data integrity and consistency

No decision has been made. The successful vendor will assist with these decisions.

12. Would unstructured data be collected as a part this objective?

Currently almost all data is structured and ingested through SQL-based transactional databases. Unstructured data is limited in scope, but should be considered as part of the evolution of datasets.

13. Follow up to Geospatial Data Question -- Are you looking to engage with data through a map-based interface?

FRA’s internal GIS team develops and maintains these applications at this time. FRA is looking to expand the use of geospatial interfaces for data collection.

14. Is this HLO focused on designing methods of data collection for known data that FRA wants to incorporate?

Or is it focused on building the

FRA expects to deal with both known data and also future data. The primary focus should be on known data but building in some flexibility in case needs evolve would be valuable.

skeleton to be able to capture any data into the EDIMS in the future? I ask because clarity on that point may help focus to get you the correct expertise.

15. Is the scope of the Data Collection limited to existing digital data or is digitization of older paper documents included in this objective? We see many Federal agencies still holding paper-based records that contain critical permanent data and they use these types of contracts to eliminate that backlog.

You would be dealing with digital data and would not be digitizing older paper documents. That step has already been done for paper documents that are received and are not part of this High Level Objective.

16. Can we leverage MuleSoft for real-time data collection and integration from external sources, such as third-party applications or government databases, into the new data warehouse

FRA is open to a variety of approaches but they must be FedRamp certified if cloud based

17. Do you plan to also do API exploration for more automated data collection, interoperability and scalability

FRA is open to techniques that enhance automated data collection, interoperability and scalability, including use of API to collect data.

18. What flexibility will FRA (and to a degree, their contractor(s)) have to adjust their data collection requirements as they continue to learn as time goes on?

FRA is flexible and open to ways to improve and enhance the data collection requirements.

19. Will you provide us with a list of the Data Sets that you're expecting the Contractor to deal with?

You can expect to work with several FRA datasets, starting with datasets currently managed under RSIS, and for that list to grow as this is applied to the entire Enterprise.

High Level Objective Five: Data Governance

1. How will data designated as CUI be handled? For CMMC Purposes. Or is that an issue at this stage?

Controlled Unclassified Information will be handled per existing cyber security requirements. The Cybersecurity Maturity Model Certification is a DOD requirement and not a DOT requirement, so there is no issue at this stage.

2. Do you have an existing Governance Board for managing data or a Change control Board in place today? What is their role today?

Yes, FRA has both a Data Architecture and Planning Review (DAPR) board for data changes and a Change Control Board (CCB) for IT/application change review. DAPR is the initial step for all new data management changes or investments.

3. How is FRA approaching data stewardship and data management? Is FRA looking for support rolling out these initiatives as part of these objectives?

Please see the bulleted items in each of the High-Level Objectives for the overall approach. This is an Enterprise approach and it needs the support of the successful vendor for rolling out all of these…