Questions for Vendors Responding to the RFI for Web Harvesting.docx
DOCX document 15 KB Posted
- Attached to
- Web Harvesting Services
- Solicitation number
- 2025-LGD-0161
- Issued by
-
Library of Congress
About this file
This document is a Request for Information (RFI) from the Library of Congress seeking vendor capabilities for web harvesting services. The RFI aims to gather market research on systematic web content harvesting, focusing on vendors' abilities to perform at-scale web harvests, handle crawler traps, manage website blocking, and provide archival crawling services. Specific questions probe vendors' technical capabilities, including distinctions between web harvesting types, browser-based crawler scales, and experience with cultural heritage or government sector web archiving.
The RFI is explicitly not a solicitation for proposals, but rather a voluntary market research exercise to understand potential commercial solutions for web content harvesting. The Library of Congress is seeking rough pricing estimates, capability statements, and feedback on a draft Statement of Work, with the ultimate goal of informing a future acquisition for systematic web content harvesting services. Vendors are encouraged to provide detailed responses about their technical approaches, crawler technologies, and strategies for managing web harvesting challenges, with the understanding that participation does not guarantee future business opportunities.
View the file
Other files
| File | Type | Posted |
|---|---|---|
| J.2_Description-of-Tech-Environment_062525_final_rfi.docx | DOCX document | |
| Web Harvesting SOW - Draft.docx | DOCX document |
On GovTribe
Work with this file
- Download the original file
- Contacts named in this file
- Similar government files
- Ask GovTribe AI about this file
Text version
Questions for Vendors Responding to RFI for Web Harvesting
1. Do you make a distinction between at-scale web harvest and high-fidelity crawling? If so, what are the distinctions?
1. Is it realistic for a single company to do both high-volume crawls and high-fidelity small-scale crawls in the scope of this contract?
1. Do you employ browser-based crawlers, if so, what is the maximum scale at which they have been demonstrably deployed?
1. Do you make a distinction between archival web crawling/web harvest, web scraping, website archiving, and social media archiving? If so, what is that distinction?
1. Are your web archiving services employed in any cultural heritage institutions (library, museum, archive, gallery or other similar institution) or government or public sector agencies? If possible, please list them.
1. We have included several new requirements and reports regarding crawler traps and websites blocking the crawler(s) (sections within 3.1 and 3.3.1.2.2) in response to developments in web security, as live websites respond to threats and AI bots. As these developments impact web archiving, would you please share your companies position and/or strategies on handling these types of barriers to web harvest?
1. Are the storage requirements clear as written in section 3.4?
File details come from the government source that posted it. Updated .