J.2_Description-of-Tech-Environment_062525_final_rfi.docx

Attachment J.2: Library of Congress Technical Environment for Web Archives

Attachment J.2: Library of Congress Technical Environment for Web Archives

The Library of Congress uses an in-house curator tool, called Digiboard, that allows the Library to manage seed URL nomination, permissions gathering, and the scoping and preparation of a seed list to send to the contractor or to crawl locally. The Library’s tool creates seed lists with scoping instructions in SURT format; seed lists are posted on a collaborative web space via a ticketing system in advance of scheduled crawls (the time frame for posting them is negotiated with Contractor), and that system is used to share other information about the status of the crawls. Once transfer has occurred, preparations for providing public access to researchers are performed (and are out outside the scope of this contract).

The Library’s web harvesting curator tools and infrastructure that support the digital lifecycle for web archives were initially developed for the inputs and outputs of Heritrix and later Brozzler for harvesting, and OpenWayback (currently version 2.4.0), and later pywb (currently version 2.8.4) and OutbackCDX (1.0.0) for access. All of these software tools are open source and are supported by the International Internet Preservation Consortium, of which the Library is a founding member (netpreserve.org). Comment by Lyon, Meghan F: Abbie asked if we still need this?

Harvested content is stored at the Library in the WARC (Web ARChive) file format.

The Library makes use of crawl reports to learn more about the contents of the crawls and to understand the depth and breadth of the crawl results. We do so by ingesting certain data elements into our in-house tools and reporting systems for quality review by the Library, and for inventorying of content. The Library has found these data elements and reports helpful to date:

· total seeds harvested and not harvested

· total hosts harvested

· total documents per host

· total documents harvested

· number of documents and size in bytes by mimetype

· HTTP response code and number of documents

· total unique documents harvested

· documents discovered but not harvested

· processed documents per second

· bandwidth in kilobytes per second

· total raw data size in bytes

· total compressed WARC size

· novel bytes

· duplicate by hash bytes

· non-modified bytes

· the duration (start and end time) of the crawl in ISO 8601 format: yyyy-mm-ddThh:mm

· number of bytes harvested per seed URL

· depth of crawl per seed URL

· Crawl information for hypertext transfer protocol (HTTP) return codes 3XX, 4XX and 5XX.

· A harvested seed list with associated URLs discovered from that seed.

· A copy of the crawl’s seed list, provided by the Library to the Contractor, shall be maintained with other crawl reports for the crawl.

· A count of WARCs written for the crawl.

· A list of PDFs harvested during the crawl.

· A report of http code, crawl status (crawled/not crawled), seed (from seed list provided) and for all 301/302 http code, the redirect.

· For multimedia content that has not replayed in the Wayback Machine, which is currently in use (such as YouTube), yet is captured during the harvests, reports that allow the Library to ensure that capture is complete have been provided and have proven helpful to the Library Quality Review processes.

· A comprehensive list of approved scopes in SURT format for each crawl.

· A custom Host data report that includes one line entry per unique host encountered during the crawl. Each line entry contains the following data:

· Total number of URIs harvested for the host

· Total number of bytes harvested for the host

· The host name string

· Number of URIs excluded for the host, due to robots.txt restriction. This number does not consider/count links from URIs specifically excluded in robots.txt.

· Number of un-crawled URIs for this host still in queue

· Number of new URIs crawled for this host since the last crawl

· Number of URIs that exhibited a hash code that matched a previous crawl, for the host. This number indicates the number of URIs that exhibited duplicate content—that is, content previously captured in the same state

· Number of bytes associated with URIs that exhibited a hash code that matched a previous crawl, for the host

· The number of URIs for the host that returned a 304 http status code

· The number of bytes associates with URIs that returned 304 http status codes for the host

An example of a Host Data Report the Library currently receives is:

[#urls] [#bytes] [host] [#robots] [#remaining] [#novel-urls] [#novel-bytes] [#dup-by-hash-urls] [#dup-by-hash-bytes] [#not-modified-urls] [#not-modified-bytes] 163912 5690495364 example.com 0 2065196 149272 5647558614 14640 42936750 0 0 147607 14152427270 2.exampletwo.com 0 11068 147273 14152222823 334 204447 0 0 126073 6885025254 www.example3.com 0 73675 71378 2498172867 54695 4386852387 0 0

p. 2 of 3