J2a - Sample Web Crawl Requirement_20251222 - (pp. 1).docx
DOCX document 16 KB Posted
- Attached to
- Web Harvesting Services - Library of Congress
- Solicitation number
- 030ADV26R0010
- Issued by
-
Library of Congress
About this file
This is an attachment to a Request for Proposals that outlines the technical requirements for a sample web crawl test that must be submitted as part of the proposal evaluation process for Library of Congress RFP 030ADV26R0010.
Offerors must conduct a one-time web crawl lasting no longer than 48 hours using a Library-provided seed list containing 100 seeds of varying type and complexity. The crawl must be performed in the proposed production environment (e.g., cloud-based or on-premises as applicable) and must be repeated for each distinct web harvesting technology the contractor proposes to use. Crawl results must be delivered in WARC format along with CDX files and reports documenting the tools and software used. The estimated size of the sample crawl data and reports must be communicated via email to the Contract Specialist (cdaly@loc.gov) and Contracting Officer (jzwa@loc.gov) no later than 5:00 PM Eastern Time on February 2, 2026. Files shall be transferred via SFTP using transfer instructions provided by the Library's Office of the Chief Information Officer and Program representatives. Upon completion of transfer, offerors must confirm receipt with the Contract Specialist and Contracting Officer. While offerors are not required to make results accessible in their proposed access tool during Library review, they may be required to demonstrate the access tool using the crawl results as part of the proposal evaluation.
View the file
Other files
Show all 24
On GovTribe
Work with this file
- Download the original file
- Contacts named in this file
- Similar government files
- Ask GovTribe AI about this file
Text version
Attachment J2a: Sample Web Crawl Requirement (12/22/2025) – page 1 of 1 Library of Congress Request for Proposals (RFP) 030ADV26R0010
Instructions:
Using a Library-provided seed list (Attachment J2b Sample Web Crawl Seed List) containing 100 seeds of varying type and complexity (plus scopes), perform a one-time crawl lasting no longer than 48 total hours. Per Library permissions policies for web archives, do not respect robots.txt.
The technical test crawl must be conducted in the environment proposed for the program, e.g. if the Contractor proposes crawling from the cloud, the test crawl must be conducted in the cloud rather than on internal servers. The technical test must be repeated for each distinct web harvest technology (i.e. crawler) proposed for utilization in the Contractor’s technical approach.
The results of the crawl shall be delivered to the Library in WARC format for the Library’s analysis, along with CDX files, and any reports generated for the crawl. Reports are outlined in SOW section C.3.3.1.2.2. Offeror’s report shall provide a brief description of the tool(s)/software used to complete the test. When the sample crawl is complete, the Offeror shall indicate to Library staff the size, or estimated size, of the sample crawl data and reports via email to the Contract Specialist (cdaly@loc.gov) and Contracting Officer (jzwa@loc.gov). This information must be received no later than 5:00pm eastern time on February 2, 2026, to enable Program representatives and Contracts Teams to prepare for transfer of content to the Library for review.
Files shall be transferred via SSH File Transfer Protocol (SFTP). The Library shall provide the Offeror submitting sample crawl files transfer instructions per the Office of the Chief Information Officer (OCIO) and Program representatives. On completion of the transfer, the Offeror must contact the Contract Specialist and Contracting Officer to confirm that all data was received.
The Offeror is not required to make the results accessible in their access tool for Library review. However, Offerors may be required to demonstrate their access tool using the results of this crawl (Section C.3.2).
File details come from the government source that posted it. Updated .