J8 - SAMPLE Task Order SOW - 20251212 (pp. 4).docx
DOCX document 40 KB Posted
- Attached to
- Web Harvesting Services - Library of Congress
- Solicitation number
- 030ADV26R0010
- Issued by
-
Library of Congress
About this file
This is a Statement of Work for a sample task order under a Web Harvesting IDIQ contract with the Library of Congress. The Library requires web harvesting services to capture digital content at scale from specified seed lists, provide temporary access for quality review, and transfer archived content to the Library for preservation and public access. The contractor must conduct four categories of crawls over an approximately 11-month period of performance, with a two-week final quality assurance and data transfer period at the end.
The specific crawl requirements include: 46 weekly crawls capturing 1,700 seed URLs per crawl with variable data limits; 11 monthly crawls capturing 4,000 seed URLs per crawl estimated to run four weeks each; 6-10 extended crawls capturing 10,000 seed URLs per crawl and running 6-12 weeks each; and 26 weekly crawls dedicated to U.S. Election 2026 content starting with approximately 100 seed URLs and expanding to 1,200-1,500 seeds by election day (November 5, 2026), then continuing through December 2026. The total estimated data collection across all crawl types is 350-700 terabytes. The Contracting Officer Representative will provide initial seed lists within five business days of task order start, with updates provided no later than 24 hours before each crawl begins. Crawl reports must be delivered within five days of completion, and invoicing will be monthly based on the exact number of terabytes delivered, inspected, and accepted by the COR in writing. Quality control procedures will follow the terms outlined in the parent IDIQ contract, with deduplication required at the start of each crawl category to establish baseline data for subsequent crawl comparisons.
View the file
Other files
Show all 24
On GovTribe
Work with this file
- Download the original file
- Contacts named in this file
- Similar government files
- Ask GovTribe AI about this file
Text version
Attachment J8 - SAMPLE Task Order SOW - 20251212 (pp. 4) STATEMENT OF WORK: Web Harvesting: Event and Thematic Collections
C.1 Background
The Library of Congress (Library) requires contract support to enable the systematic harvesting of content from the web based on instructions from Library staff, to provide temporary access to the content and required crawl reports for quality review, and to enable transfer of content to the Library for preservation and public access. This is a task order under [TBD] Web Harvesting IDIQ. The terms and conditions of the the IDIQ contract are applicable to this task order.
C.2 Scope
The Library’s Web Archiving Program manages many aspects of the web archiving workflow for adding to Library of Congress collections, web harvesting services are required for capture of web content at scale. Harvested content shall be transferred to the Library for preservation and access by Library staff.
C.3 Requirements
The contractor shall capture the web content in the seed lists provided by the Library at the frequencies outlined below in accordance with Section C of the IDIQ. The Contracting Officer Representative (COR) will provide an initial seed list within five (5) business days of task order period of performance start date. Updates to the seed list will be provided no later than 24 hours prior to the start of each crawl.
The contractor shall capture specified web content from the Library-provided seed lists at frequencies described in the requirements outlined below. The period of performance for this task order will be identified at the time of award. Crawls under this task will occur as identified in the task order at the time of award. Each task order will have an approximate two week period at the end of the period of performance for final quality assurance, acceptance, and transfer of data to the Library.
The Contractor shall collect the estimated Number of Terabytes, 350-700 TB, in the following four categories of crawls:
C.3.1 Weekly crawls
The weekly crawls will capture specified content at a weekly frequency for seed URLs where content is changing frequently.
The contractor must conduct crawls in accordance with section C of the contract as defined below:
A. Numbers of Crawls: 46 (one per week) B. Estimated Number of Seed URLs per crawl: 1,700.
C. Data limit for crawl: Variable based on size of initial undeduplicated and subsequent crawls.
D. Length of crawls: One week, or until agreed-upon data caps have been met.
E. Deduplication: Undeduplicated crawl must be performed at start of the task order to dedupe against during the remaining crawls. Undeduplicated crawl will have a data limit applied.
F. Robots.txt: Ignore, except in cases where instructed otherwise by the Library in the seed list.
G. WARC and CDX filenaming convention: TBD upon start of IDIQ contract.
C.3.2 Monthly crawls
These crawls are to capture content with a monthly frequency where content is changing less often.
The contractor must conduct crawls in accordance with section C of the contract as defined below:
A. Numbers of Crawls: 11 (one per month for 11 months) B. Estimated Number of Seed URLs per crawl: 4,000 C. Data limit per crawl: Variable based on size of initial undeduplicated and subsequent crawls.
D. Length of crawls: Estimated 4 weeks, or until agreed-upon data caps have been met.
E. Deduplication: Undeduplicated crawl must be performed at start of task order to dedupe against during the remaining crawls.
F. Robots.txt: Ignore, except in cases where instructed otherwise by the Library in the seed list.
G. WARC and CDX filenaming convention: TBD upon start of IDIQ contract.
C.3.3 Extended crawls
This task is to perform in-depth capture of priority content, including very large sites and others requiring more in-depth capture than is possible in lesser frequent crawls.
The Library schedules extended crawls that run until a pre-determined data or time limit is reached. Start dates are to be determined on the time of order award.
The contractor must conduct crawls in accordance with section C of the contract as defined below:
A. Numbers of Crawls: 6-10 B. Estimated Number of Seed URLs per crawl: 10,000 C. Length of crawls: Estimated 6-12 weeks, or until agreed-upon data caps have been met.
D. Data limit per crawl: Variable based on size of initial undeduplicated and subsequent crawls.
E. Deduplication: Undeduplicated crawl must be performed at start of task order for each extended type to dedupe against during the remaining crawls.
F. Robots.txt: Ignore, except in cases where instructed otherwise by the Library.
G. WARC and CDX filenaming conventions: TBD upon start of IDIQ contract.
C.3.5 US Election 2026 Weekly crawls
The U.S. Election 2026 weekly crawls will capture specified content at a weekly frequency for seed URLs where content is changing frequently (ie. campaign websites).
The contractor must conduct crawls in accordance with section C of the contract as defined below:
A. Number of crawls: 26 (one per week thru the end of December) B. Estimated Number of Seed URLs per crawl: The crawls will start with approximately 100 seed URLs and will be incrementally added to up through Election day November 5, 2026 with a maximum of about 1,200-1,500 seeds. After election day, the seed list will remain stable or decrease until the crawls are stopped at the end of December 2026.
C. Data limit for crawl: Variable based on size of initial undeduplicated and subsequent crawls.
D. Length of crawls: One week, or until agreed-upon data caps have been met.
E. Deduplication: Undeduplicated crawl must be performed at the first crawl to dedupe against during the remaining crawls.
F. Robots.txt: ignore, except in cases where instructed otherwise by the Library G. WARC and CDX filenaming convention: TBD upon start of IDIQ contract.
C.4 Deliverables
Reports required per C.3.3.1 of the IDIQ contract must be provided no later than 5 days after each crawl is completed. Quality-assurance results, as per C.3.2 and C.9 of the IDIQ contract, must be submitted in accordance with a Quality Assurance Plan that has been approved by the Library COR.
Unless otherwise required and mutually agreed upon in writing by the contractor and the COR, the contractor shall invoice monthly for the exact number of terabytes of data delivered, inspected, and fully accepted in writing by the COR.
C.5 Government Furnished information:
Seed lists are the property of the Library and are solely for the Library's collection purposes.
C.6 Quality Control Quality Control shall follow the requirements outlined in Section C.4.2 of IDIQ contract unless otherwise described.
File details come from the government source that posted it. Updated .