Web Harvesting SOW - Draft.docx

WEB HARVESTING SERVICES IDIQ

1. OVERVIEW / BACKGROUND. The Library of Congress (Library) requires contract support to enable the systematic harvesting of content at-scale from the web, based on instructions from Library staff, to provide temporary access to the content and required crawl reports for quality review, and to enable transfer of content to the Library for preservation and public access.

Since 2000, the Library has collected and preserved harvested web content related to a variety of thematic web and event-based topics, based on Library collecting policies. The web archives are an important component of the Library’s born digital collections. The harvesting of selected websites for the Library’s collections supports the Library’s Strategic Goal to build and enrich Library collections and content to serve Congress and America’s communities.

The Library uses an in-house curatorial tool called Digiboard to manage the web archiving program, including to export seed lists to deliver to a Contractor for crawling. Selection of materials for this program is conducted by Library staff who are subject and language experts. The Library Web Archiving Program staff are the technical liaisons between the Contractor performing harvesting services and Library staff engaged in selection and management of the content after it’s been harvested and delivered to the Library.

Library crawls typically include a large quantity of seed URLs per crawl. Due to scale, the Library’s Web Archiving program and approach to collecting have historically amounted to a large-scale crawling effort, organizing selection around thematic collections while crawling in large “buckets” based on desired frequency of harvest. The number of distinct seed URLs in each ‘frequency bucket’ range from hundreds to thousands per seed list. Seeds are scheduled for crawling at regular intervals or “frequency”, which may include weekly, monthly, and quarterly, however past iterations have included twice-daily and yearly.

Content targeted for archiving may include entire domains, specific subdomains or paths, portions of websites focused on a particular topic, or single documents (such as PDFs). Content in the web archives is published in the United States and in multitude of other countries, with varying languages. The Library’s web archives follow a permissions-based model; careful curation occurs by Library staff, and the Library creates scoping instructions at the seed-level to ensure crawled content remains in scope.

2. SCOPE

While the Library’s Web Archiving Program manages many aspects of the web archiving workflow for adding to Library of Congress collections, web harvesting services are required for capture of web content at scale. Instructions about what to harvest are based on seed lists provided by the Library to the Contractor prior to each crawl’s start date, updated with new or changed seeds prior to each new crawl. Scoping instructions and crawling rules are provided to the Contractor by the Library with each seed list to ensure that content is captured by the Contractor in accordance with the Library’s permissions and collecting policies. The Contractor must provide access to Web Archiving program staff for quality assurance to accept the results of the crawls and store a copy of the data during the period of each task order (task orders issued typically cover about a year’s worth of crawls). Harvested content is required to be regularly transferred to the Library by the Contractor for preservation and staff and researcher access by the Library.

Specific performance areas include:

· Web content capture according to scheduling and deduplication of crawls as specified by the Library (3.1)

· Access to captured content for Library quality review (3.2)

· Reports (3.3)

· Web content hosting and storage (3.4)

· Transfer of content to the Library (3.5)

· Increasing politeness settings on a seed or domain-specific basis

· custom regex to exclude problematic URL patterns, such as infinitely repeating directories or self-generating 404s

· site specific crawler configuration settings (e.g. sheet rules like queue assignments in Heritrix)

For each crawl, the Contractor must provide the following reports within 5 days of the completion of each crawl. The Contractor must supply sample crawl reports and file-naming conventions at the start of the first task order under this IDIQ and confirm format and data elements. The Contractor must notify Library of any format changes to the crawl reports prior to implementation of changes.

In the event that multiple types of crawlers are used, the Contractor must supply sample reports from each type of crawler at the start of the first task order under this IDIQ. The Contractor must ensure that the data in all reports are complete and accurate. Reports from distinct crawlers must be in parity with each other, in terms of formatting and data elements.

Reports must be viewable in the vendor’s system to enable quality review by the Library, and transferred to the Library (see C.5):

(See Attachment J.2 - Technical Environment section for details on how the Library uses crawl report data in its inspection, and Library QA processes.)

· A record of what was crawled, with a list of seeds crawled and a list of hosts crawled

· A record of the breadth and depth of the crawl, i.e.

· how much from each seed list was captured

· number of resources and size per host

· number of resources still left in queue at the end of the crawl

· amount of data archived per host and per seed

· amount of new data vs. duplicated data per host and per seed

· lists of URLs not crawled because they were out of scope

· number of successfully crawled seeds

· number of redirected seeds

· number of seeds not crawled (if any)

· number of seeds queued (if any)

· number of seeds blocked (if any)

· A copy of the seed list provided by the Library to the Contractor for the crawl.

· All crawler specifications, patches, scripts, and configurations (e.g., order file, special rules for the crawler) used for generating the crawl.

· A report of the total number of bytes harvested per seed URL in that crawl, in rank order

· A report of the total depth the crawler reached from each seed URL in that crawl, in rank order

· A list of http response codes and status (e.g. ‘crawled’ or ‘not crawled’) and redirects for each seed crawled.

· A report of crawler traps encountered by the crawler during routine crawls. This data will be used by the Library and the Contractor to write negative SURTs or exclude-regex to limit the amount of out-of-scope content harvested, ensuring the crawlers have sufficient resources for crawling in-scope content, and to prevent over-crawling nominated websites.

· A report on bot-blocks identified at the seed and resource level, including lists of domains routinely blocking the crawler(s), the frequency of blocks in the context of routine crawls, and the source of repeat blocks.

3.3.1.3. All available crawl logs as generated by the crawler(s).

3.3.3 The Contractor must provide reports on any errors encountered during integrity monitoring for hosted data as described in Section 3.4, including checksum errors and operational status, and any remedies taken.

3.4.1.2. The Contractor must ensure that the infrastructure for crawling, storing, and providing access to content provides documented physical security, electronic security, and operational practices that represent standard industry practices for commercial hosting providers.

The Contractor is required to maintain and store the access and backup copies, and any crawl reports, until the task order is complete, and all content has been transferred to the Library per section 3.5.

3.5.4.1 NETWORK TRANSFER REQUIREMENTS

This section applies if the Contractor’s method of transfer is using a network:

Deliverables will vary by task and will be specified within each task order. Each task order will identify deliverables, deadlines and milestones for the work required, including number of maximum terabytes ordered, type and frequency of crawls and number of crawls, estimated number of seeds per crawl, length of crawls and any data caps, and other specifications based on the specific order. All reports and deliverables shall be delivered to the COR.

4.2. QUALITY CONTROL.

The Contractor shall develop and maintain an effective quality control program (QCP) to ensure services are performed in accordance with this contract. The Contractor shall develop and implement procedures to identify, prevent, and ensure non-recurrence of defective services and to ensure that requested content has been harvested. The plan should accommodate robust procedures to identify, track and resolve issues in the event of repeated failure to harvest nominated content.

The Contractor’s quality control program is the means by which it assures itself that its work complies with the requirement of the contract. The QCP should be delivered within 30 days after the contract award to the COR. The plan must list the contents of the quality assurance report(s) that the Contractor proposes to provide for each completed crawl and steps taken to ensure quality.

In the creation of a QCP, the Contractor may consider how the plan may address the following questions:

· What are the quality objectives or goals for this contract?

· What are the processes and procedures that will be employed to control quality of the products and services delivered under the contract?

· What are the roles and responsibilities for both the Contractor and the Library about these QC processes and procedures?

· What are the quality standards that will be enforced for the contract?

· What are the quality measures or metrics that will used to demonstrate that the quality standards have been met or exceeded?

· How will monitoring and inspection be performed for the contract?

· How will quality failures be prevented?

· How will the Contractor ensure that remediated failures do not recur?

· How will the Contractor identify ways to improve the efficiency or effectiveness of its services?

· How will the Contractor identify and mitigate risks to quality performance?

· How will the Contractor and the Library collaborate to resolve disputes about quality?

After acceptance of the quality control plan the Contractor shall receive the contracting officer’s acceptance in writing of any proposed change to its QC system.

6.

6.1. KEY PERSONNEL (if applicable). The following positions are considered key personnel:

Program Manager/alternate program manager: The Contractor shall provide a program manager who shall be responsible for the performance of the work. The name of this person and an alternate who shall act for the Contractor when the manager is absent shall be designated in writing to the contracting officer. The program manager or alternate shall have full authority to act for the Contractor on all contract matters relating to daily operation of this contract. The program manager or alternate shall be available between 9:00 a.m. to 5:30p.m. Monday thru Friday except Federal holidays or when the government facility is closed for administrative reasons. The Program Manager must have at least 5 years relevant experience managing projects or programs.

Crawl Engineer: The Contractor shall provide a crawl engineer who is responsible for the management of and reporting on the harvesting activities. The crawl engineer must have at least five years relevant experience managing web harvesting crawls.

Quality Assurance Lead: The Contractor shall provide a quality assurance lead who is responsible for ensuring that the QAP is followed by any Contractor staff. The Quality Assurance Lead must have at least 2 years relevant experience performing quality assurance on web archives.

6.2 OTHER (If applicable):

DELETION OF CAPTURED CONTENT. At the end of each task order, upon confirmation from the Library that all copies of captured content and crawl logs and reports have been successfully transferred to the Library, the Contractor must delete all copies of captured content stored by the Contractor.

7. PHASE IN /PHASE OUT PERIOD. To minimize any decreases in productivity and to prevent possible negative impacts on additional services, the Contractor must have personnel on board, during the thirty (30) day phase in/ phase out periods. During the phase in period, the Contractor must become familiar with performance requirements in order to commence full performance of services on the contract start date.

(End of Performance Work Statement / Statement of Work)