Q and A - August 1.pdf

1. Is there an incumbent on this contract? If so, please provide the incumbent name, current contract number, Period of performance, value of the contract?

GOVERNMENT RESPONSE:

Incumbent: GUIDEHOUSE INC.

Contract Number: LCCRS23P0004 Period of Performance: 9/29/2023 – 9/28/2024 Value: $1,785,946.44

2. What is the allocated budget for this solicita�on?

GOVERNMENT RESPONSE: Less than $1,900,000.00

3. Is this a follow-on contract?

GOVERNMENT RESPONSE: Yes, this is a compe��ve recompete solicita�on for a reoccurring requirement at the Library.

4. Would CRS consider having all of the training sessions done remotely to save cost?

GOVERNMENT RESPONSE: No

5. What current data analy�c tools and pla�orms does CRS u�lize?

GOVERNMENT RESPONSE: CRS is u�lizing the AWS pla�orm and some of the analy�cs tools within that pla�orm. Addi�onally, CRS has access to PowerBI and some access to Tableau.

Analysts in CRS are also using R, Python, and Stata for some projects.

6. Under Key Personnel sec�on, it is men�oned that we need Program Manager and The Program

Manager, we need to know if we are supposed to provide resumes for both, or it is a single posi�on.

GOVERNMENT RESPONSE: (The) Program Manager is a single posi�on. Note, there is a requirement for an Alternate Program Manager as described in Sec�on C.5.1 and H.7.

7. In SECTION B - SUPPLIES OR SERVICES AND PRICE/COSTS, the quan�ty required for Item 1 is 1 LOT, for item 2 it is 10 MO and for item 3 it is 5 EA.

GOVERNMENT RESPONSE: Correct. 10 MO is in reference to Ten Months, 5 EA is in reference to the Five Models listed in Sec�on C.

8. Regarding the Atachment J1 - Pricing sheet, we want to know can we change it according to our style, like can we add the Labor Mix, or we need to provide Labor Mix in separate excel sheet?

Addi�onally, do we need to men�on Labor Mix in Staffing volume also?

GOVERNMENT RESPONSE: The Library would like to see the Labor Mix in both, the Price Proposal and an unpriced version in the Staffing Volume.

9. The RFP states that the contractor shall consider CRS's currently available data, tools, and architecture, as well as what is available in the broader marketplace to support quan�ta�ve analysis and modeling for policy analysis. Could the government clarify what tools and technologies are currently used by CRS?

GOVERNMENT RESPONSE: See ques�on 5.

10. Page 7 of the RFP document states “In FY 2023, CRS ini�ated a pilot to develop 5 web-based data models as a proof of concept. These models primarily support the Domes�c Social Policy Division, Foreign Affairs, Defense and Trade Division and administra�ve ac�vi�es.” Please provide details of the contractor that developed the 5 web-based data models?

GOVERNMENT RESPONSE: See ques�on 1.

11. Page 7 of the RFP document states “In FY 2023, CRS ini�ated a pilot to develop 5 web-based data

Division, Foreign Affairs, Defense and Trade Division and administra�ve ac�vi�es.” Please clarify if these proof of concepts are for the same data models being developed as part of this contract or are they different ones.

GOVERNMENT RESPONSE: Three of the new data models in sec�on 3.7 have some component of building on the exis�ng models—Model 3, Model 4, and Model 5. The other two models are completely new. The models ge�ng support in sec�on 3.9 are the proof of concept models.

12. Page 7 of the RFP document states “In FY 2023, CRS ini�ated a pilot to develop 5 web-based data

Division, Foreign Affairs, Defense and Trade Division and administra�ve ac�vi�es.” If these proof-of-concept data models are different from the ones being requested on this contract, how are they related to the models developed as part of this contract?

GOVERNMENT RESPONSE: See ques�on 11.

13. Will the government allow us to leverage open-source technologies?

GOVERNMENT RESPONSE: Specific open-source technologies would require sign off by LOC IT, but are not completely off the table for use.

14. Could the government clarify the sources from which CRS currently ingests health insurance pricing data for Model 3? Assump�on Model 3 sec�on 3.7

GOVERNMENT RESPONSE:

• Large MRFs from Insurance provider websites. Updated monthly

• NPPES data for providers, updated frequently for new providers

15. Could the government clarify the sources from which CRS currently ingests data for Model 4?

Assump�on Model 4 sec�on 3.7 GOVERNMENT RESPONSE: Data is ingested from CRS data files.

16. Could you please provide the link for the DOD Budget Jus�fica�ons Q&A Portal?

GOVERNMENT RESPONSE: This is what we are asking the contractor to build—there is no exis�ng link.

17. Could you please clarify the following ques�ons for Model 1?

a. What is the backend persistence for comments? Specifically, what data beyond the basic fields of comment, date, and commenter is captured (e.g., metadata such as IP address, and browser informa�on)?

GOVERNMENT RESPONSE: For model 1 in sec�on 3.7, this model is not yet developed.

The intent is to capture user sen�ment, key word analysis and summary informa�on for comments.

b. What limita�ons, if any, are there on the data that is captured?

GOVERNMENT RESPONSE: The informa�on is public, so there should be no limita�ons.

c. What languages of commenters are currently supported? Addi�onally, which languages should be supported to meet the pla�orm's requirements?

GOVERNMENT RESPONSE: There appear to be no limita�ons on the languages that can be used on Regula�ons.gov. However, most of the comments are expected to be in English.

18. Could you please clarify Model 2 is there a code base to be shared in advance for Model 2?

GOVERNMENT RESPONSE: For Model 2 in sec�on 3.7, the expecta�on is that the contractor would be crea�ng this model, so there is no exis�ng code at LOC.

19. Could you please clarify the following ques�ons for Model 3? Assump�on Model 3 sec�on 3.7

a. Where can the following plan details be found, or is this level of detail not required?

Coverage levels, premiums, deduc�bles, co-pays, out-of-pocket maximums.

GOVERNMENT RESPONSE:

• Large MRFs from Insurance provider websites. Updated monthly

• NPPES data for providers, updated frequently for new providers

b. Where can network informa�on be found, or is this level of detail not required? Provider directories, hospital affilia�ons.

GOVERNMENT RESPONSE: See 19a.

c. Where can drug formularies be found, or is this level of detail not required? Covered medica�ons, �ers, and cost-sharing.

GOVERNMENT RESPONSE: See 19a

d. Where can customer service informa�on be found, or is this level of detail not required?

Contact details, claims processing, appeals.

GOVERNMENT RESPONSE: See 19a.

20. Could you please clarify the following ques�ons for Model 4? Assump�on Model 4 in 3.7

a. What is the exis�ng persistence of the data?

GOVERNMENT RESPONSE: CRS would like to be able to create visualiza�ons of certain scenarios. The data should be retained long enough to allow for visualiza�ons.

b. How are exis�ng models defined? Are they defined using hard-coded procedures or some other method?

GOVERNMENT RESPONSE: The model is coded in Python but uses some uploaded economic data.

c. How extensible should plans be? What level of flexibility and scalability is required for future enhancements and expansions?

GOVERNMENT RESPONSE: The model should have the flexibility to incorporate a wide range of policy modifica�ons, changes in economic benchmarks, and borrower demographic characteris�cs.

21. Could you please clarify the following ques�ons for Model 5? Assump�on Model 5 sec�on 3.7

a. Data Sources: Where does the model currently pull data from?

GOVERNMENT RESPONSE:

• Dash-1 files: Office of the Under Secretary of Defense (Comptroller)

• DOD Comptroller (defense-wide budget justification documents):

https://comptroller.defense.gov/BudgetMaterials/FY2025budgetjustification.aspx

• Department of the Army: https://www.asafm.army.mil/Budget-Materials/

• Department of the Navy: https://www.secnav.navy.mil/fmc/Pages/Fiscal-Year-2025.aspx

• Department of the Air Force: https://www.saffm.hq.af.mil/FM-Resources/Budget/

b. Data Structure: How is the data organized? Is the data stored in a rela�onal database, data warehouse, CSV files, or another format?

GOVERNMENT RESPONSE: The data are excel files and pdf files.

c. Model Architecture: What technologies and algorithms are used for data processing and analysis?

GOVERNMENT RESPONSE: Model architecture for PDF inges�on is in design phase.

Currently manual inges�on from websites into the AWS environment. The AWS tool QuickSight is then used for analysis and visualiza�on.

d. Current Func�onali�es: What ques�ons can the model currently answer?

GOVERNMENT RESPONSE: The current model is a database of consolidated informa�on which can be manipulated by analysts. A key word search is also being developed, but will not answer specific ques�ons.

22. Model 1: Will we be provided any private access to "regula�ons.gov" datasets, APIs, or otherwise?

GOVERNMENT RESPONSE: No. Contractor should expect to use the public data.

https://comptroller.defense.gov/Budget-Materials/ https://comptroller.defense.gov/BudgetMaterials/FY2025budgetjustification.aspx https://www.asafm.army.mil/Budget-Materials/ https://www.secnav.navy.mil/fmc/Pages/Fiscal-Year-2025.aspx https://www.saffm.hq.af.mil/FM-Resources/Budget/

23. Model 2: In what environment (e.g. AWS, on-prem) is this model deployed?

GOVERNMENT RESPONSE: Model 2 in sec�on 3.7 is not created yet. I would be expected to be in AWS at the end of the solicited contract.

24. Model 2: Are there specific impact analyses to target as a requirement under this contract?

GOVERNMENT RESPONSE: No. The model is simply a calculator. CRS economists will determine what impact analyses to target once the tool is complete and veted by the CRS tax team based on its research needs and policy proposal requiring analysis.

25. Model 2: If data will be made available to us to enhance the Individual Income Tax Calculator, what is the total volume of the data in gigabytes?

GOVERNMENT RESPONSE: The contractor would have to determine what data was necessary to create the model. Current best es�mate is less than 5 GB.

26. Model 2: If data will be made available to us to enhance the Individual Income Tax Calculator, will any of the data feature Personally Iden�fiable Informa�on (PII)?

GOVERNMENT RESPONSE: No.

27. Model 2: Please provide 3 examples of impact analysis supported by the current tools.

GOVERNMENT RESPONSE: CRS only has one in-house tax model (aka tool), which is its marginal effec�ve tax rate model that is used to study the impact of tax changes on business investment.

The expecta�on is that the contractor will create Model 2 in sec�on 3.7—this is not an enhancement or con�nua�on of work on a current tool.

28. Model 2: What is the total quan�ty of impact analyses which must be supported by the updated model at the end of this contract?

GOVERNMENT RESPONSE: The total quan�ty of impact analyses must accommodate analyses of the current tax code and proposed changes.

29. Model 2: What raw data was used to create the exis�ng Tax-Calculator by the Policy Simula�on Library?

GOVERNMENT RESPONSE: The contractor would be crea�ng the model. They could contact the Policy Simula�on Library to see if they are willing to share how their model was created and use that as their base. Some data is available at htps://taxcalc.pslmodels.org/.

30. Model 2: What data types (i.e. tabular, �me series, images, pdfs) were used to create the exis�ng Tax-Calculator used by the Policy Simula�on Library?

GOVERNMENT RESPONSE: The contractor would be crea�ng the model.

31. Model 3: On average, how o�en does CRS currently ingest updated informa�on from health insurers?

GOVERNMENT RESPONSE: There could be new informa�on as o�en as monthly.

https://taxcalc.pslmodels.org/

32. Model 3: Will the Health Insurance Pricing data visualiza�on tool and/or web-based dashboards be made available externally to Congress or the general public?

GOVERNMENT RESPONSE: The visualiza�ons maybe shared, but there’s not an expecta�on that anyone outside CRS will work within the models.

33. Model 4: If data will be made available to us to enhance the Student Loan Calculator, what is the total volume of the data in gigabytes?

GOVERNMENT RESPONSE: This is a calculator and large volumes of data are not stored.

34. Model 4: If data will be made available to us to enhance the Student Loan Calculator, will any of the data feature Personally Iden�fiable Informa�on (PII)?

35. Model 4: What raw data was used to create the exis�ng Student Loan Calculator Project?

GOVERNMENT RESPONSE: Code of a similar product already in use at CRS and some data files on economic condi�ons.

36. Model 4: What data types (i.e. tabular, �me series, images, pdfs) were used to create the exis�ng Student Loan Calculator Project?

GOVERNMENT RESPONSE: There are some data files uploaded on income trajectories and HHS poverty files.

37. Model 4: Will the Student Loan Calculator data visualiza�on tool and/or web-based dashboards be made available externally to Congress or the general public?

GOVERNMENT RESPONSE: The visualiza�ons maybe shared, but there’s not an expecta�on that anyone outside CRS will work within the models.

38. Model 5: In what environment (e.g. AWS, on-prem) is this model deployed?

GOVERNMENT RESPONSE: AWS

39. Model 5: Will the users be querying this data (i.e. SQL), and performing keyword search?

GOVERNMENT RESPONSE: The request is to create a tool that would allow ques�ons to be asked and data returned to answer those ques�ons. The contractor is free to do this how they like.

40. Model 5: By rough es�mate, how many ques�ons might be asked to the DOD Budget Jus�fica�ons Q&A Portal on a day of par�cularly high traffic?

GOVERNMENT RESPONSE: Fewer than 20 ques�ons.

41. What are the performance metrics by which a CRS analyst's success is measured? If there aren't any current metrics, what kind of benefits does CRS expect the 5 models to deliver by end of engagement?

GOVERNMENT RESPONSE: Analyst performance metrics are not a factor in the current contract.

CRS expects the models to enhance our service to Congress by allowing more data analysis.

42. What are the types of posi�ons or roles of the 25-50 atendees for the training seminars?

GOVERNMENT RESPONSE: Most of the staff at CRS are analysts. There are also atorneys, librarians and admin staff.

43. How o�en will CRS analysts use models, data visualiza�ons, or dashboards during the development phase?

GOVERNMENT RESPONSE: During development CRS analysts will only be using models for tes�ng.

44. Is there any remaining work to produc�onize the 5 pre-exis�ng models?

GOVERNMENT RESPONSE: Yes, the models are s�ll being created but are expected to be func�onal by the end of the current contract.

45. Outside of fluent model usage, what new knowledge or skills should CRS analysts acquire by the conclusion of our contract as a result of our training sessions?

GOVERNMENT RESPONSE: The training sessions on the models are not part of 3.4 or 3.5 that would be included in 3.7.1. The expecta�on of 3.4 and 3.5 would be exposure to other sta�s�cal tools/methods. Examples of training offered this last year include Introduc�on to Python, data management, and overview of data visualiza�on tools.

46. How does CRS expect analysts to apply these skills a�er training is complete? What is the expecta�on of what training is to accomplish?

GOVERNMENT RESPONSE: CRS understands that the training isn’t long enough to provide exper�se. The goal of the training sessions is to give staff exposure to tools/methods they have never used and allow them have some hands-on exposure to the tools/methods.

47. Does the Library expect a breakdown of the labor mix for every deliverable, e.g. hours and rates for each Project Status Report? The template in atachment J1 only indicates price per unit.

GOVERNMENT RESPONSE: Yes the Library would like to see the labor mix breakdown if possible.

48. Would the government consider increasing the page limit of the technical approach (Volume 1) to 15 pages?

GOVERNMENT RESPONSE: Yes, the page limit is increased to 15 pages.

49. Would the government please include the following two deliverables "SOW 3.1 Kick-Off Mee�ng" and "SOW 3.9 - Coaching and Support on 5 Exis�ng Models" in the 030ADV24R0053 - Atachment J1 in the pricing sheet?

GOVERNMENT RESPONSE: No, the Library does not plan to include the deliverable "SOW 3.1 Kick-Off Mee�ng" in Atachment J1. "SOW 3.9 - Coaching and Support on 5 Exis�ng Models" will be included in Atachment J1.

50. For model 1 to 5, what is the frequency of data extrac�on and model refresh?

a. If yes, Who is responsible for model refresh?

GOVERNMENT RESPONSE: There is an expecta�on that data would be refreshed and there may be the need for contractor support if the refresh fails.

51. What is the status of data quality, data readiness, data availability of the test data for the new models? Is the contractor expected to help with any of these tasks for each model?

GOVERNMENT RESPONSE: It is expected that by the end of the current contract, the data will be available and ready. The new contractor may be asked to help with new data inges�on.

52. What is the size of the current incumbent team - FTEs for exis�ng 5 model development and implementa�on?

GOVERNMENT RESPONSE: The current team has 13 members who are data scien�sts or cloud engineers/architects. However, all members of the team don’t seem to be devoted full �me to the effort and some seem to be on board to support training efforts over data model crea�on.

53. Is there Model Observability in place now for exis�ng models?

54. What is the typical model tes�ng and approval �meline at LOC?

GOVERNMENT RESPONSE: In general, tes�ng should be expected to take approximately 2 weeks once final models are provided.

55. What is the role of the steering commitee in the approvals of each new model and/or analy�cal tools?

GOVERNMENT RESPONSE: The tools are tested/reviewed by analyst who will ul�mately work with the tools.

56. Can the 3.7 in Payment schedule be broken up for payment on delivery of each model?

GOVERNMENT RESPONSE: Yes, models should be independently priced and payment will be made when a model is func�onal.

57. What is the long term hos�ng vision of the organiza�on? When will this be decided? If it is already decided, can the LOC share the details?

GOVERNMENT RESPONSE: Hos�ng will be moved to the LOC AWS environment.

58. What is the current tech stack on which the current models are developed and deployed?

GOVERNMENT RESPONSE: Models are currently being developed in contractor AWS space and no models are fully deployed in LOC environments.

59. What are the analy�cal tools besides R, Python that they CRS users are using currently?

GOVERNMENT RESPONSE: Some AWS tools are being used such as QuickSight. SAS and STATA are also used—though not in the current models. There is also access to Power BI and some access to Tableau.

60. Are there any known capability challenges or technical limita�ons that exist in the current architecture (regarding 1) data processing, 2) analy�cs, 3) model development, the LOC would like to see addressed?

GOVERNMENT RESPONSE: The biggest challenge is the size of the data sets for the Health Care model.

61. Are the data repositories (data sources for models, analy�cs and repor�ng (visualiza�on)) for models available for produc�on implementa�on?

a. If not, when will they be available?

GOVERNMENT RESPONSE: It is expected that model ATO will be completed by November. Once completed, contractors will have full access to the exis�ng models.

62. What is the current architecture for data connec�vity between LOC and incumbent hos�ng environment for models, analy�cs and visualiza�on tools?

GOVERNMENT RESPONSE: While models are in development, staff tes�ng the models are logging into the contractor’s AWS environment.

63. Has the LOC performed data quality assessment of the data sources for the new models for opera�onaliza�on?

a. Can LOC share the assessment details?

GOVERNMENT RESPONSE: Data for new models (sec�on 3.7) is all public informa�on.

With the excep�on of the health care data, all other data sources are fairly straight forward.

64. Is the LOC sa�sfied with the current data quality of the data in the databases for visualiza�on and for performing analysis?

GOVERNMENT RESPONSE: Underlying models are s�ll in development.

65. How is the data fed into the model? Is there a direct connec�vity between the LOC and the Contractor hosted environment for data feeds (Model inputs/outputs) as well as for performing analysis and developing visualiza�on tools?

GOVERNMENT RESPONSE: For current models all data is uploaded—there are currently no direct links beyond public websites for any of the models.

66. How are the current models being used or opera�onalized?

GOVERNMENT RESPONSE: The current models are s�ll being created.

67. Are there systems in LOC that access the model now?

a. If not, will there be systems accessing and running models in the future?

GOVERNMENT RESPONSE: Systems don’t access models. Models generally access public informa�on or get a push of data from an LOC system that is uploaded. Any other inputs are done via user.

68. Will the contractor be responsible for opera�onalizing the models?

GOVERNMENT RESPONSE: Yes, the contractor is responsible for crea�ng the models discussed in 3.7 and making them func�onal.

69. What is the current per month hos�ng cost for exis�ng models? Compute Usage, No of users...

Or do we do a pass through?

GOVERNMENT RESPONSE: Current models will be hosted in LOC environment and costs are paid directly to hos�ng contractor (AWS).

70. Can LOC share the proposed transi�on out plan from the current incumbent? Specifically hos�ng rights, licensing transfers, model/data applica�ons training session, documenta�on, code and expected �meline for transi�on?

GOVERNMENT RESPONSE: The exis�ng models will be hosted in the LOC AWS—contractor will be working on those models within the LOC environment. System documenta�on is in a centralized SharePoint site that the new contractor will have access to.

71. Is there already a preferred technology stack currently in the LOC network that will be used to host the Model?

GOVERNMENT RESPONSE: Models are expected to be hosted in Library’s AWS environment.

72. Is FedRamp Moderate is your only compliance requirement?

GOVERNMENT RESPONSE: Yes, though it is expected that final models will have to be moved to the LOC AWS environment.