Web Harvesting Services
Closed Pre-Solicitation Posted
- Solicitation number
- 2025-LGD-0161
- Agency
-
Library of Congress
- Responses due
- Set-aside
- No set-aside
Opportunity facts
- NAICS code
- 519210 Libraries and Archives
- PSC
- 9999 Miscellaneous Items
- Place of performance
- Washington, District of Columbia, United States
Notice details come from SAM.gov. Updated .
About this opportunity
The Library of Congress is conducting a Request for Information (RFI) to gather market research on web harvesting services that will enable systematic, at-scale harvesting of web content for preservation and public access. This pre-solicitation notice seeks contractor capabilities to perform web crawling based on Library staff instructions, provide temporary access for quality review, generate comprehensive crawl reports, and facilitate secure transfer of harvested content to the Library for long-term preservation. The contract structure will be an Indefinite Delivery/Indefinite Quantity (IDIQ) arrangement requiring contractors to use U.S.-based servers, implement polite crawling practices, support various crawl frequencies, capture password-protected content, and deliver content in WARC file formats. Key personnel requirements include a Program Manager with 5+ years of experience, a Crawl Engineer with 5+ years of web harvesting experience, and a Quality Assurance Lead with 2+ years of relevant experience. The Library is seeking capability statements, rough order of magnitude pricing estimates, and feedback on the draft Statement of Work by July 25, 2025, with participation being voluntary and no guarantee of future business.
This opportunity does not include any set-aside designations and is open to all eligible contractors. No incumbent contractors or current service providers are identified in the documentation, suggesting this may be a new requirement or a significant restructuring of existing services. The solicitation does not specify award values, budget ranges, or contract ceiling amounts, as this RFI is purely for market research purposes with no immediate contract award planned. The Library's current technical environment utilizes open-source tools including Heritrix and Brozzler for harvesting, and OpenWayback, pywb, and OutbackCDX for access, along with an in-house curator tool called Digiboard for managing URL nomination and seed list preparation. The RFI emphasizes that no expenses will be reimbursed for contractor participation and that this notice constitutes market research only, not an official solicitation for proposals.
Notice text
This is a Request for Information (RFI) to obtain capability statements and potential feedback for an upcoming Library of Congress (LOC) requirement.
Specifically, the Library of Congress (Library) requires contract support to enable the systematic harvesting of content at-scale from the web, based on instructions from Library staff, to provide temporary access to the content and required crawl reports for quality review, and to enable transfer of content to the Library for preservation and public access.
In order to obtain this information, the Library is seeking capabilities statements, rough orders of magnitude estimates, input on the Statement of Work, or any other information your business would care to provide in order to help the Library understand the possible solutions available on the commercial marketplace.
All information received shall be reviewed to help the Library inform its possible future acquisition decisions relating to this requirement. Participation in this RFI is voluntary only and is no guarantee of future business.
As an RFI, this is NOT an official solicitation. The Library is not seeking to obtain official proposals at this time as a result of this notice. The Library is seeking outside information from potential offerors as a way to perform supplemental market research. No contract shall be awarded as a result of this RFI. This notice solely constitutes market research. Under NO circumstances shall any expenses be repaid as a result of contractor engagement with RFI this notice.
Offerors are encouraged to ask questions or provide feedback so that the LOC may enhance its understanding to reflect a more accurate business environment. Attached is a draft Statement of Work. Offerors are encouraged to provide feedback on this document as well, and all comments/questions/suggestions shall be considered (this is not a guarantee that they will be answered or reflected in the final solicitation).
Finally, rough orders of magnitude in terms of pricing is also being asked for. Please note, these would be estimates only and serve only the purposes of market research. These are NOT official quotes, but are considered to be high level rough ballpark estimates of other marketplace solutions.
Files
| File | Type | Posted |
|---|---|---|
| Web Harvesting SOW - Draft.docx | DOCX document | |
| J.2_Description-of-Tech-Environment_062525_final_rfi.docx | DOCX document | |
| Questions for Vendors Responding to the RFI for Web Harvesting.docx | DOCX document |
On GovTribe
Work this opportunity
- Track it in your pipeline
- Find teaming partners
- Similar opportunities
- Ask GovTribe AI about this opportunity