030ADV26R0010.pdf

PDF 451 KB Posted

Attached to
Web Harvesting Services - Library of Congress
Solicitation number
030ADV26R0010
Issued by

About this file

This is a Solicitation/Contract/Order for Commercial Products and Commercial Services (Standard Form 1449) issued by the Library of Congress for Web Harvesting Services.

The Library of Congress seeks an Indefinite-Delivery, Indefinite-Quantity (IDIQ) contract for web harvesting services with a minimum guarantee of $300,000 and a maximum value of $15,000,000 over a five-year base period (June 1, 2026 to May 31, 2031). The contractor must systematically harvest web content at scale based on Library instructions, provide temporary access to harvested content, generate required crawl reports for quality review, and enable content transfer to the Library for preservation and public access. Key requirements include performing crawls using multiple crawl technologies (standard and browser-based), maintaining two copies of harvested content, providing a web-based access tool for Library quality review within five business days of crawl completion, packaging content in valid WARC format with CDX indexes, and ensuring all crawling infrastructure is physically located in the United States. The contractor must deliver content within five days of Library acceptance and maintain system availability with maximum annual downtime of 20 business days during Library business hours. Minimum order is $300,000 and maximum single order is $5,000,000. The offer due date is February 24, 2026 at 5:00 PM EST. Proposals must be submitted electronically to the Contracting Officer and must include four volumes: Technical Approach (30 pages maximum), Sample Web Crawl results, Corporate Experience and Capabilities, and Pricing. Evaluation will be conducted using best-value tradeoff methodology, with technical factors being equally weighted and significantly more important than price. Key personnel include a Program Manager (5 years experience required), Crawl Engineer (5 years experience), and Quality Assurance Lead (2 years experience). Ordering years run from June 1 through May 31 for each fiscal year, with firm fixed-price task orders based on estimated annual needs. The contract incorporates FAR clauses, Library-specific provisions regarding generative AI prohibition, information security, contractor employee fitness requirements, and various federal compliance certifications.

View the file

Other files

File Type Posted
J3 - Past Performance Questionnaire - PPQ - (pp. 2)_20260224.docx DOCX document
J9 - QA_030ADV26R0010_20260224 (pp. 7).pdf PDF
030ADV26R0010 0007.pdf PDF
J9 - QA_030ADV26R0010_20260219 (pp. 7).pdf PDF
030ADV26R0010 0006.pdf PDF
030ADV26R0010 0005.pdf PDF
030ADV26R0010 0004.pdf PDF
J9 - QA_030ADV26R0010_20260213 (pp. 7).pdf PDF
030ADV26R0010 0003.pdf PDF
J9 - QA_030ADV26R0010_20260203 (pp. 6).pdf PDF
J3 - Past Performance Questionnaire - PPQ - (pp. 2)_20260128.docx DOCX document
030ADV26R0010 0002.pdf PDF
J9 - QA_030ADV26R0010_20260128 (p. 1).pdf PDF
Amendment 1 - 030ADV26R0010 0001.pdf PDF
Amendment 1 - J2a - Sample Web Crawl Requirement_20260123 - (pp. 1).docx DOCX document
J2a - Sample Web Crawl Requirement_20251222 - (pp. 1).docx DOCX document
J2b - Sample_Web_Crawl_Seed_List_20250917 - (pp. 13).docx DOCX document
J1 - Description-of-Tech-Environment_20250912 - (pp. 3).docx DOCX document
J4 - Price Schedule_WebHarvesting - (pp. 1).xlsx XLSX spreadsheet
J3 - Past Performance Questionnaire - PPQ - (pp. 2).docx DOCX document
J6 - Configuring Crawl Jobs - Heritrix 3_20260115 (pp. 22).pdf PDF
J7 - ISO 28500_2017 WARC file format_20260115 (pp. 2).pdf PDF
J8 - SAMPLE Task Order SOW - 20251212 (pp. 4).docx DOCX document
J5 - Web Archiving FAQ_20260115 (pp. 4).pdf PDF
Show all 24

On GovTribe

Work with this file

  • Download the original file
  • Contacts named in this file
  • Similar government files
  • Ask GovTribe AI about this file

Text version

PAGE 1 OF1. REQUISITION NO.

2. CONTRACT NO. 3. AWARD/EFFECTIVE DATE 4. ORDER NO. 5. SOLICITATION NUMBER 6. SOLICITATION ISSUE DATE

a. NAME b. TELEPHONE NO. (No Collect Calls) 8. OFFER DUE DATE/LOCAL

TIME

9. ISSUED BY CODE 10. THIS ACQUISITION IS UNRESTRICTED OR SET ASIDE: % FOR:

SMALL BUSINESS

HUBZONE SMALL

BUSINESS

SERVICE-DISABLED

VETERAN-OWNED

SMALL BUSINESS

WOMEN-OWNED SMALL BUSINESS

(WOSB) ELIGIBLE UNDER THE WOMEN-OWNED

SMALL BUSINESS PROGRAM

EDWOSB

8(A)

NAICS:

SIZE STANDARD:

11. DELIVERY FOR FOB DESTINA-

TION UNLESS BLOCK IS

MARKED

SEE SCHEDULE

12. DISCOUNT TERMS

13a. THIS CONTRACT IS A

RATED ORDER UNDER

DPAS (15 CFR 700)

13b. RATING

14. METHOD OF SOLICITATION

RFQ IFB RFP

15. DELIVER TO CODE 16. ADMINISTERED BY CODE

17a. CONTRACTOR/OFFEROR CODE FACILITY CODE 18a. PAYMENT WILL BE MADE BY CODE

TELEPHONE NO. UEI: EFT:

PHONE: FAX:

17b. CHECK IF REMITTANCE IS DIFFERENT AND PUT SUCH ADDRESS IN OFFER

18b. SUBMIT INVOICES TO ADDRESS SHOWN IN BLOCK 18a UNLESS BLOCK BELOW IS CHECKED

SEE ADDENDUM

19. 20. 21. 22. 23. 24.

ITEM NO. SCHEDULE OF SUPPLIES/SERVICES QUANTITY UNIT UNIT PRICE AMOUNT

(Use Reverse and/or Attach Additional Sheets as Necessary)

25. ACCOUNTING AND APPROPRIATION DATA 26. TOTAL AWARD AMOUNT (For Govt. Use Only)

27a. SOLICITATION INCORPORATES BY REFERENCE FAR 52.212-1, 52.212-4. FAR 52.212-3 AND 52.212-5 ARE ATTACHED. ADDENDA ARE ARE NOT ATTACHED.

27b. CONTRACT/PURCHASE ORDER INCORPORATES BY REFERENCE FAR 52.212-4. FAR 52.212-5 IS ATTACHED. ADDENDA ARE ARE NOT ATTACHED

28. CONTRACTOR IS REQUIRED TO SIGN THIS DOCUMENT AND RETURN _______________ 29. AWARD OF CONTRACT: REF. ___________________________________ OFFER COPIES TO ISSUING OFFICE. CONTRACTOR AGREES TO FURNISH AND DATED ________________________________. YOUR OFFER ON SOLICITATION DELIVER ALL ITEMS SET FORTH OR OTHERWISE IDENTIFIED ABOVE AND ON ANY (BLOCK 5), INCLUDING ANY ADDITIONS OR CHANGES WHICH ARE ADDITIONAL SHEETS SUBJECT TO THE TERMS AND CONDITIONS SPECIFIED SET FORTH HEREIN IS ACCEPTED AS TO ITEMS:

30a. SIGNATURE OF OFFEROR/CONTRACTOR 31a. UNITED STATES OF AMERICA (SIGNATURE OF CONTRACTING OFFICER)

30b. NAME AND TITLE OF SIGNER (TYPE OR PRINT) 30c. DATE SIGNED 31b. NAME OF CONTRACTING OFFICER (TYPE OR PRINT) 31c. DATE SIGNED

AUTHORIZED FOR LOCAL REPRODUCTION (REV. NOV 2021)

PREVIOUS EDITION IS NOT USABLE Prescribed by GSA - FAR (48 CFR) 53.212

7. FOR SOLICITATION

INFORMATION CALL:

STANDARD FORM 1449

SOLICITATION/CONTRACT/ORDER FOR COMMERCIAL PRODUCTS AND COMMERCIAL SERVICES

OFFEROR TO COMPLETE BLOCKS 12, 17, 23, 24, & 30

See SCHEDULE

030ADV26R0010

Zwahlen, Jennifer (202) 707-0925 02-24-2026

17:00 EST

Contracts and Grants Directorate Library of Congress

101 Independence Ave. SE LA 325 Washington DC 20540-9411

X

N/A

X

DELIVERY

See Sections D&F for delivery instructions.

Contracts & Grants Directorate Library of Congress

LA 325

101 Independence Av SE Washington DC 20540-9411

Submit Invoices through https://www.ipp.gov Washington DC 20540

See CONTINUATION Page

See CONTINUATION Page

X X

Zwahlen, Jennifer

030ADV26R0010

Table of Contents

SECTION A

A.1 SF 1449 SOLICITATION/CONTRACT/ORDER FOR COMMERCIAL PRODUCTS AND COMMERCIAL SERVICES

SECTION B - SUPPLIES OR SERVICES AND PRICE/COSTS

B.1 PRICE/COST SCHEDULE

ITEM INFORMATION

B.2 LC52.212-2 SUPPLEMENTAL LICENSE AGREEMENTS (JUL 2017)

B.3 LC52.216-3 CONTRACT TYPE (JUN 2016)

B.4. SUPPLIES OR SERVICES AND PRICING

SECTION C - DESCRIPTION/SPECIFICATIONS/STATEMENT OF WORK

C. STATEMENT OF WORK

SECTION F - DELIVERIES OR PERFORMANCE

F.1 LC52.215-5 PLACE AND PERIOD OF PERFORMANCE (JAN 2019)

SECTION G - CONTRACT ADMINISTRATION DATA

G.1 LC52.201-3 CONTRACT ADMINISTRATION (AUG 2015)

G.2 LC52.232-1 LIBRARY OF CONGRESS INVOICE INSTRUCTIONS (DEC 2022)

G.3 LC52.232-2 SCHEDULE OF PAYMENTS (JUN 2015)

SECTION H - SPECIAL CONTRACT REQUIREMENTS

H.1 Use of Generative Artificial Intelligence (AI)

H.1 LC52.201-1 CONTRACTING OFFICER'S REPRESENTATIVE (COR) APPOINTMENT AND AUTHORITY (JAN 2016)

H.2 LC52.203-1 CONTRACTOR PUBLICITY (OCT 2013)

H.3 LC52.204-1 COLLECTIONS SECURITY (DEC 2013)

H.4 LC52.204-3 INFORMATION SYSTEMS SECURITY (JUL 2024)

H.5 LC52.204-4 CONTRACTOR EMPLOYEE FITNESS (SEP 2024)

H.6 LC52.237-1 KEY PERSONNEL (AUG 2014)

PART II - CONTRACT CLAUSES

SECTION I - CONTRACT CLAUSES

I.1 52.252-2 CLAUSES INCORPORATED BY REFERENCE (FEB 1998)

I.2 52.212-5 CONTRACT TERMS AND CONDITIONS REQUIRED TO IMPLEMENT STATUTES OR EXECUTIVE

ORDERS—COMMERCIAL PRODUCTS AND COMMERCIAL SERVICES (OCT 2025)

I.3 52.216-18 ORDERING (AUG 2020)

I.4 52.216-19 ORDER LIMITATIONS (OCT 1995)

030ADV26R0010

I.5 52.216-22 INDEFINITE QUANTITY (OCT 1995)

I.6 LC52.212-4 CONTRACT TERMS AND CONDITIONS - COMMERCIAL ITEMS (DEV) (FEB 2016)

I.7 LC52.212-5 CONTRACT TERMS AND CONDITIONS REQUIRED TO IMPLEMENT STATUTES OR EXECUTIVE

ORDERS - COMMERCIAL ITEMS (DEV) (MAY 2015)

I.8 LC52.216-1 ORDER PLACEMENT – SINGLE AWARD INDEFINITE DELIVERY, INDEFINITE QUANTITY (FEB 2016)

I.9 LC52.232-25 PROMPT PAYMENT (DEV) (JUL 2013)

I.10 LC52.239-1 INFORMATION SECURITY (JUL 2025)

I.11 LC52.233-1 52.233-1 (DEV) DISPUTES (MAY 2014)

PART III - LIST OF DOCUMENTS, EXHIBITS AND OTHER ATTACHMENTS

SECTION J - LIST OF ATTACHMENTS

PART IV - REPRESENTATIONS AND INSTRUCTIONS

SECTION K - REPRESENTATIONS, CERTIFICATIONS AND OTHER STATEMENTS OF OFFERORS

K.1 52.212-3 OFFEROR REPRESENTATIONS AND CERTIFICATIONS—COMMERCIAL PRODUCTS AND

COMMERCIAL SERVICES (OCT 2025)

SECTION L - INSTRUCTIONS, CONDITIONS, AND NOTICES TO OFFERORS

L.1 LC52.215-1 PROPOSAL SUBMISSION INSTRUCTIONS - BEST VALUE TRADE-OFF (JUL 2016)

L.2 LC52.233-2 PROTESTS TO THE LIBRARY OF CONGRESS (DEC 2013)

SECTION M - EVALUATION FACTORS FOR AWARD

M.1 LC52.215-2 EVALUATION - BEST VALUE TRADE-OFF (FEB 2020)

030ADV26R0010 Section B

SECTION B - SUPPLIES OR SERVICES AND PRICE/COSTS

B.1 PRICE/COST SCHEDULE

ITEM INFORMATION

ITEM

NUMBER

DESCRIPTION OF

SUPPLIES/SERVICES QUANTITY UNIT UNIT PRICE AMOUNT

0001 5.00 YR _______________

Web Harvesting Services Contract Period: Base POP Begin: 06-01-2026 POP End: 05-31-2031

GRAND TOTAL _______________

B.2 LC52.212-2 SUPPLEMENTAL LICENSE AGREEMENTS (JUL 2017)

The Library does not agree to licensing agreements or any other provisions that attempt to: (1) replace an order as the governing contract; (2) establish new or supplemental payment terms; or, (3) affect the legal relationships of the parties.

Notwithstanding any clauses in this contract or order, or the contractual vehicles against which the order is placed, that establish order of precedence, the rights and obligations provided by this contract or order take precedence over the rights and obligations set forth in any licensing agreements provided to the Library.

The Library’s incorporation of a licensing agreement into this contract or order excludes all terms of the licensing agreement that contravene, modify or conflict with the terms of 36 C.F.R. 701.7 available at: https://www.ecfr.gov/cgi-bin/text-idx?SID=87a865ce2bd109313a951a3d477895d8&mc=true&node=se36.3.701_17&rgn=div8.

(End of Clause)

B.3 LC52.216-3 CONTRACT TYPE (JUN 2016)

This is an Indefinite-Delivery, Indefinite-Quantity (IDIQ) contract with firm-fixed-price Task Orders.

(End of Clause)

B.4. SUPPLIES OR SERVICES AND PRICING

B.4.1 Minimum and Maximum Amounts

Minimum: $300,000.00

030ADV26R0010 Section B

Maximum: $15,000,000.00

The Library of Congress will order the specified minimum amount under this IDIQ contract. Specific requirements will be identified in individual task orders issued under this contract. During the life of this contract, the Government may order any combination of services established in Section C for any quantity. The cumulative value of all task orders issued under this IDIQ contract shall not exceed the maximum amount specified under this IDIQ for the life of the contract. The maximum amount applies to this contract and the total of all orders placed under this contract. The Government is under no obligation to place further orders after the initial minimum guarantee established at $300,000.00.

B.4.2 Ordering Years and Unit Prices

This IDIQ contract is established with Ordering Years instead of Option Years and, as such, annual exercising of contract options is not required. The Ordering Years for this Contract are as follows:

All Supplies and Services

Ordering Year I: 06/01/2026 – 05/31/2027

Ordering Year II: 06/01/2027 – 05/31/2028

Ordering Year III: 06/01/2028 – 05/31/2029

Ordering Year IV: 06/01/2029 – 05/31/2030

Ordering Year V: 06/01/2030 – 05/31/2031

Unit prices for all deliverables shall be established, by Ordering Year, in Attachment J4 – Pricing Schedule. Order prices shall correspond with the Ordering Year prices that are active at the start of the period of performance, despite which Ordering Year most of the work takes place in. No mixed-year pricing will be utilized. For example, an order with a period of performance start date of 08/10/2026 which ends 08/09/2027 would be priced entirely using Ordering Year I prices even though a portion of the order overlaps with Ordering Year II.

030ADV26R0010 Section C

SECTION C - DESCRIPTION/SPECIFICATIONS/STATEMENT OF WORK

C. STATEMENT OF WORK

C.1. OVERVIEW / BACKGROUND

The Library of Congress (Library) requires contract support to enable the systematic harvesting of content at-scale from the web, based on instructions from Library staff, to provide temporary access to the content and required crawl reports for quality review, and to enable transfer of content to the Library for preservation and public access.

Since 2000, the Library has collected and preserved harvested web content related to a variety of thematic web and event-based topics, based on Library collecting policies. The web archives are an important component of the Library’s born digital collections. The harvesting of selected websites for the Library’s collections supports the Library’s Strategic Goal to build and enrich Library collections and content to serve Congress and America’s communities.

The Library uses an in-house curatorial tool called Digiboard to manage the web archiving program, including to export seed lists to deliver to a Contractor for crawling. Selection of materials for this program is conducted by Library staff who are subject experts. The Library Web Archiving Program staff are the technical liaisons between the Contractor performing harvesting services and Library staff engaged in selection and management of the content after it’s been harvested and delivered to the Library.

Library crawls typically include a large quantity of seed URLs per crawl. Due to scale, the Library’s Web Archiving program and approach to collecting have historically amounted to a large-scale crawling effort, organizing selection around thematic collections while crawling in large “buckets” based on desired frequency of harvest. The number of distinct seed URLs in each ‘frequency bucket’ range from hundreds to thousands per seed list. Seeds are scheduled for crawling at regular intervals or “frequency”, which may include weekly, monthly, and quarterly; however, past iterations have included twice-daily and yearly.

Content targeted for archiving may include entire domains, specific subdomains or paths, portions of websites focused on a particular topic, or single documents (such as PDFs). Content in the web archives is published in the United States and in multitude of other countries, with varying languages. The Library’s web archives follow a permissions-based model; careful curation occurs by Library staff, and the Library creates scoping instructions at the seed-level to ensure crawled content remains in scope. The Library previously acquired public social media pages as seed URLs when nominated by Recommending Officers at the Library, or as scopes when linked from home page of the targeted domain.

We discontinued this practice in 2021, around the time of broad implementation of log-in walls by social media platforms, but continue to monitor developments around social media archiving.

Contractors performing this work to-date have utilized multiple crawl technologies to support web harvest operations.

Most commonly, the program has benefited from a large, standard (e.g. Heritrix) crawl, and a smaller browser-based crawl. While the percentage of this split has varied, a rough estimate of the split assuming around 16,500 seed URLs across all crawls is 70/30 standard to browser-based. A smaller sub-set of websites in the 30 percent designated to browser-based crawling may benefit from more customized crawling, however due to the construct of past programs this has not been feasible.

See the Library's Rights and Access policy at this link: https://www.loc.gov/collections/american-music-industry-web-archive/about-this-collection/rights-and-access/

C.2. SCOPE

While the Library’s Web Archiving Program manages many aspects of the web archiving workflow for adding to Library of Congress collections, web harvesting services are required for capture of web content at scale. Harvested content shall be transferred to the Library for preservation and access by Library staff.

C.3. REQUIREMENTS

This Statement of Work (SOW) contains several weblinks to Library and industry resources. The Library may alert the contractor when a change is made to the link and/or its contents, but it is the contractor's responsibility to monitor these links and the information and standards provided therein. The contractor shall contact the Library for an updated link if one provided in this SOW becomes inaccessible or obsolete and its replacement is not immediately apparent.

C.3.1 GENERAL CRAWL REQUIREMENTS

3.1.1 The Contractor must perform crawls to capture content identified by the Library according to the Library’s specifications. For each crawl, the Library will provide the Contractor with formatted seed lists which identify the seed URLs to be captured, scoping instructions, robots.txt instructions, and other rules for that crawl. The Library reserves the right to make minor revisions to the seed list up to 24 hours in advance of the scheduled crawl.

3.1.2 The Contractor must perform crawls according to crawl schedules specified by the Library in accordance with task orders, which shall be issued under this contract. Each task order will establish the crawl schedule and crawl frequencies at the start of each task. Crawls may have varying start dates, frequencies, durations, and data limits. The Library will provide seed lists for the Contractor. Seed lists are the property of the Library and are solely for the Library’s collection purposes.

3.1.3 The Contractor must complete crawls within a timeframe, which shall be defined in each order, sufficient to maintain the specified frequency, while employing politeness factors.

3.1.4 The Contractor must allow for robots.txt exclusions either to be ignored or to be respected for seeds within a crawl.

3.1.5 The Contractor must allow for deduplication of content against baseline crawls, storing only data that has been modified or newly discovered when compared with data previously captured for and stored on behalf of the Library.

3.1.6 The Contractor must enable as comprehensive of a capture as possible given current tools and stated capabilities of the Contractor. The Contractor must enable capture of a variety of digital objects from the web required to render the identified seeds as they were at the time of capture. Digital objects may include, but are not limited to, HTML, images, PDFs, multimedia including but not limited to audio and video files, any embedded content, and other essential components necessary for accurately replicating a webpage. When playing back the archived content, the Contractor’s priority shall be to recreate an experience that mirrors what a user would have encountered when visiting the live site at the specific time of capture.

3.1.7 The Contractor shall provide flexible approaches to enable the capture content at the level of depth and breadth and per scoping instructions for each seed, including highly complex sites with multiple paths and many links from the primary seed URL. Novel approaches to organizing the crawls and strategies which employ multiple crawlers are welcome to maximize successful capture of targeted content.

3.1.8 The contractor must crawl websites politely with all employed crawlers, in order to minimize server impact, as described in Attachment J5 – Web-Archiving-FAQ. Such ‘politeness’ settings are explained in Heritrix, as an example (see Attachment J6 - Configuring Crawl Jobs - Heritrix 3). Politeness settings will be reviewed and agreed upon at the start of the contract but may be reassessed if need arises.

3.1.9 The Contractor must enable strategies for crawling seed lists, which may include thousands of seed URLs and scoping instructions, in such a way as to minimize server impact on websites, web hosts and resources being harvested, including in cases where different sections of the same website may be represented as numerous independent seeds in the same crawl. Strategies shall include, but are not limited to:

Increase politeness settings on a seed or domain-specific basis Custom regex to exclude problematic URL patterns, such as infinitely repeating directories or self-generating

404s Site specific crawler configuration settings (e.g. sheet rules like queue assignments in Heritrix in Attachment J6)

3.1.10 The Contractor shall utilize one or more types of web crawlers to accomplish at-scale harvest. Additional crawlers may be utilized to accomplish high-fidelity capture of all or part of each seed list.

3.1.10.1 Additional crawlers, if browser-based, must include rate-limiting or other polite crawling capabilities (see 3.1.3 and 3.1.8).

3.1.10.2 The Contractor shall suggest strategies to accomplish harvest of all or sub-sets of seed lists using multiple crawl technologies, e.g. splitting seed lists into smaller or single-seed crawls. The Contractor should provide sustainable means of routing and tracking seed URLs designated for such crawls. Contractor must account for details in the Quality Control Plan (see 4.2).

3.1.11 The Library recognizes that the web is complex and constantly evolving, and approaches to crawling at-scale are ever-changing. To allow the Library to best anticipate quality of results of the crawls, the Contractor must inform the Library of significant changes to how the crawler is able to capture identified content, or crawling approaches that affects (both positively and negatively) the quality of results or ability to capture content identified by the Library.

Additionally, the Contractor shall test and use of new and improved tools to capture content are encouraged, for instance when harvesting challenging content. The Contractor must notify the Library prior to testing and implementation of any new crawling tools used to capture content for the Library. The Contractor must provide technical specifications and descriptive information (such as an overview of what it does, how it works, and how it differs from the current in-use tools) about the new proposed tool and the Library will review the proposal and provide approval or feedback in 10 business days. The Library reserves the right to reject a proposed tool

3.1.12 The Contractor must be flexible in scoping and capping different types of content to best capture seeds identified by the Library for harvesting; sometimes this includes making changes to running crawls. Changes may be made at the direction of the Library or if the contractor needs to stop and adjust a running crawl for reasons that may include interruption of service, removal of a URL from the crawl queue, or other server impacts on the live site. Per the below 3.1.15, if a change or pause is initiated by the contractor, the contractor must inform the Library within 24 hours of issues, interruptions, and resolutions.

3.1.13 The Contractor must capture password-protected content when provided with credentials by the Library.

3.1.14 The Contractor must run additional crawls, if necessary, to harvest materials missed during a crawl. Such crawls shall occur only as directed by the Library on a limited basis.

3.1.15 The Contractor must monitor each crawl and, within 24 hours, notify the Library and take action to resolve any issues related to crawler behavior or interruption of service.

3.1.16 The Contractor must package captured content in valid WARC (Web ARChive) files (see format in Attachment J7

– ISO 28500_2017 WARC file format) for transfer to the Library, accompanied by valid, 11-field CDX indexes for all resources in the WARC files.

3.1.17 The Contractor must use servers physically located in the United States for all crawling (i.e., capture of content) that occurs under this contract. If the Contractor provides cloud crawling services, the Contractor must utilize U.S. based data centers. There is no restriction on server location for other services provided under this contract.

3.1.18 The Contractor shall provide a web-based communication tool and mechanism (such as a ticketing system), available via a secure login, for exchange of seed lists between the Library and the Contractor, reporting of issues, quality assurance inquiries and responses, tracking transfer of content, and other general communications.

3.1.19 The Contractor must stay up to date with requirements in the larger field of bots and crawlers and observe practices that will enable them to remain ‘verified’ or off ‘bad bot’ lists. The Contractor should routinely provide data on whether these conditions are met and the extent of such ‘bot protection’ or anti-bot blocking’s impact on harvest quality. This is a complex, new and evolving field which, when encountered, may significantly detract from crawl success.

3.1.20 The Contractor must provide valid user agents and static IP addresses for the crawlers at the start of each task order.

C.3.2 ACCESS FOR LIBRARY QUALITY REVIEW

3.2.1 The Contractor must provide an access tool on a Contractor-provided system by which Library staff can review the results of the crawls performed by the Contractor prior to transfer of the content to the Library. The Contractor must restrict access to Library staff and the Contractor only. Access must not be available to the public or Library users not on the list provided by the COR (see 3.2.3.1).

3.2.2 The Contractor must make the results of each crawl accessible to the Library within five (5) business days of completion via the Contractor-provided access tool.

3.2.3 Access Tool Requirements

3.2.3.1 The Contractor’s access tool must allow for up to 50 simultaneous sessions (restricted by IP address to Library premises only or by password protection) to use the access interface. The Library will provide a list of staff users who are allowed to have access to the vendor’s access tool.

3.2.3.2 The access tool must replay archived content and support search, retrieval, and display by URL.

3.2.3.3. The access tool must limit the display of content to harvested content (i.e., preventing content from the “live” web from displaying), and must clearly distinguish between archived and live content.

3.2.4 The Contractor must ensure that all content captured is fully functional within the accepted limitations when displayed via the access tools.

3.2.5 The Contractor must provide an action plan to monitor, report to the Library, and remedy breaks in access service. When it is within Contractor’s ability to schedule maintenance outages, outages must be scheduled outside of Library business hours (6:30am – 6:00pm EST Monday-Friday) if possible, and Library staff must be notified 24 hours in advance of any planned outages.

3.2.5 The Contractor must provide the most current, stable version of its access tool to the Library for the Library’s review of crawled content.

C.3.3 REPORTS

3.3.1 REPORTS REQUIREMENTS FOR EACH CRAWL

For each crawl, the Contractor must provide the following reports within 5 days of the completion of each crawl. The Contractor must supply sample crawl reports and file-naming conventions at the start of the first task order under this IDIQ and confirm format and data elements. The Contractor must notify the Library of any format changes to the crawl reports before the Contractor implements changes.

In the event that multiple types of crawlers are used, the Contractor must supply sample reports from each type of crawler at the start of the first task order under this IDIQ. The Contractor must ensure that the data in all reports are complete and accurate. Reports from distinct crawl technologies must be in parity with each other, in terms of formatting and data elements

Reports must be viewable in the vendor’s system to enable quality review by the Library, and transferred to the Library (see C.5):

3.3.1.1 Contractor shall provide quality assurance results as specified in the Quality Assurance Plan (See C.4.)

3.3.1.2. Contractor shall provide statistical information about the crawl.

(See Attachment J.1 - Technical Environment section for details on how the Library uses crawl report data in its inspection, and Library QA processes.)

3.3.1.2.1 The Contractor must provide reports to the Library in ASCII text and xml format.

3.3.1.2.2 The Contractor’s reports must contain the following information (some reports may be generated by the crawler, some reports are custom reports requested for quality control purposes):

A record of what was crawled, with a list of seeds crawled and a list of hosts crawled A record of the breadth and depth of the crawl, i.e.

o how much from each seed list was captured o number of resources and size per host o number of resources still left in queue at the end of the crawl o amount of data archived per host and per seed o amount of new data vs. duplicated data per host and per seed o lists of URLs not crawled because they were out of scope o number of successfully crawled seeds o number of redirected seeds o number of seeds not crawled (if any) o number of seeds queued (if any) o number of seeds blocked (if any)

A copy of the seed list provided by the Library to the Contractor for the crawl.

All crawler specifications, patches, scripts, and configuration files (e.g., crawler beans, order file, special rules for the crawler) used for generating the crawl.

A report of the total number of bytes harvested per seed URL in that crawl, in rank order A report of the total depth the crawler reached from each seed URL in that crawl, in rank order A list of http response codes and status (e.g. ‘crawled’ or ‘not crawled’) and redirects for each seed crawled.

A report of crawler traps encountered by the crawler during routine crawls.

A report on bot-blocks identified at the seed and resource level, including lists of domains routinely blocking the crawler(s), the frequency of blocks in the context of routine crawls, and the source of repeat blocks.

3.3.1.3. The Contractor shall provide all available crawl logs as generated by the crawler(s).

3.3.2 The Contractor must provide reports on any errors encountered during integrity monitoring for hosted data as described in Section 3.4, including checksum errors and operational status, and any remedies taken.

C.3.3.3 Library Ownership of Reports

The Library will own the rights to all reports created under this contract. The contractor agrees not to use any data that is not available in the public domain, nor information included in reports for the Library, with generative AI nor in its work with any other individual not authorized to perform work under this contract.

C.3.4 VENDOR HOSTING AND STORAGE INFRASTRUCTURE

3.4.1. The Contractor must maintain a reliable and secure data storage and maintenance infrastructure capable of storing all web content harvested within the task orders issued under this contract incorporating the following minimum capabilities as outlined below:

3.4.1.1. The Contractor must store and maintain two copies of web content harvested under the scope of each task order: one for access by Library staff (see 3.2), and one for backup to ensure that data is not lost between the time of capture and the time it is transferred to the Library.

3.4.1.2. The Contractor must ensure that the infrastructure for crawling, storing, and providing access to content provides documented physical security, electronic security, and operational practices that represent standard industry practices for commercial hosting providers.

3.4.1.3. The Contractor must perform some file-level integrity monitoring for data hosted under the terms of this contract.

3.4.2. The Contractor shall perform fixity checks on the content when bagged and prior to transmission to the library.

3.4.2.1. The Contractor must monitor the operational status of the environment for providing access to Library content and notify the Library of any downtime.

3.4.3 Duration of Content Storage

The Contractor is required to maintain and store the access and backup copies, and any crawl reports, until the task order is complete, and all content has been transferred to the Library per section 3.5.

C.3.5 TRANSFER OF CONTENT TO THE LIBRARY

3.5.1 The Contractor must enable transfer within 5 days of harvested content to the Library after quality review and acceptance by the Library.

3.5.2 The Contractor must provide a method of verifying complete transfers of content. The Library’s preferred method for transfer is the BagIt File Packaging Format specification, a hierarchical file packaging format. See https://datatracker.ietf.org/doc/html/rfc8493 for information.

3.5.2.1 For content delivered in BagIt structure, no single bag should exceed 1 TB. Recommended size for a BagIt bag is 250-500 GB in total (e.g., 250-500 1-GB WARC files per bag)

3.5.3 The Library’s standard for a WARC file target size is ~1 GB.

3.5.4 The Library will provide one cloud storage space (e.g. S3 bucket) for the Contractor to deliver packaged content in BagIt format. The Contractor shall deliver that content over any secure internet connection, including Internet 2, using https protocols.

3.5.5. If the Contractor’s method of transfer is using a network, the following Network Transfer Requirements apply:

3.5.5.1 The Contractor must maintain machines and networks operational for transfer, as described above.

3.5.5.2 The Contractor must address and rectify transfer failures falling within its machine(s) and/or network sphere. A transfer failure is a factor that causes transfer of currently requested content to fail.

3.5.5.3 The Contractor must acknowledge a transfer failure query from the Library within 24 hours.

3.5.5.4 If the transfer failure is determined to fall within the Contractor’s machine(s) and/or network sphere, the Contractor must rectify the failure within five (5) business days.

3.5.5.5 If rectifying the failure takes more than five (5) business days, the total time required to rectify it must be counted as downtime (see 3.5.4.1.6).

3.5.5.6 The Contractor must ensure system availability for transfer of harvested content to the Library during the Library’s business hours in Eastern time. In no case must Contractor’s annual downtime exceed 20 business days or portions thereof.

3.5.5.7 In the case of unavoidable scheduled downtime, the Contractor must give the Library 24 hours’ notice.

C.4. CONTRACT PERFORMANCE

C.4.1. SCHEDULE OF EVENTS/DELIVERABLES

Each task order shall contain a schedule of deliverables. Each task order will identify the specific deliverables, deadlines and milestones for the work required; including number of maximum terabytes ordered; type and frequency of crawls and number of crawls; estimated number of seeds per crawl; length of crawls and any data caps; and other specifications based on the specific order. The Contractor shall deliver all reports and deliverables to the COR (see Section 6.2 for specific contact information.). Unless otherwise identified at the task order level, the contractor shall invoice monthly for the exact number of terabytes of data delivered, inspected, and fully accepted in writing by the COR.

Deliverables Due Date

Kick-off Meeting Within 10 days after contract award

Final Quality Control Plan Within 30 days after contract award

Reports described under 3.3.1 Within 5 days of the completion of each crawl

Content Available for Inspection by the Library (See 3.2)

Within 5 days of the completion of each crawl

Content Available for Transfer to the Library via (See 3.5)

Within 5 days of acceptance of each crawl

C.4.2. QUALITY CONTROL

The Contractor shall develop and maintain an effective quality control program (QCP) to ensure services are performed in accordance with this contract. The Contractor shall develop and implement procedures to identify, prevent, and ensure non-recurrence of defective services and to ensure that requested content has been harvested. The plan should accommodate robust procedures to identify, track and resolve issues in the event of repeated failure to harvest nominated content.

The Contractor’s quality control program is the means by which it assures itself that its work complies with the requirement of the contract. The QCP should be delivered within 30 days after the contract award to the COR. The plan must list the contents of the quality assurance report(s) that the Contractor proposes to provide for each completed crawl and steps taken to ensure quality.

In the creation of a QCP, the Contractor must consider how the plan shall address the following questions:

What are the quality objectives or goals for this contract?

What are the processes and procedures that will be employed to control quality of the products and services delivered under the contract?

What are the roles and responsibilities for both the Contractor and the Library about these QC processes and procedures?

What are the quality standards that will be enforced for the contract?

What are the quality measures or metrics that will used to demonstrate that the quality standards have been met or exceeded?

How will monitoring and inspection be performed for the contract?

How will quality failures be prevented?

How will the Contractor ensure that remediated failures do not recur?

How will the Contractor identify ways to improve the efficiency or effectiveness of its services?

How will the Contractor identify and mitigate risks to quality performance?

How will the Contractor and the Library collaborate to resolve disputes about quality?

After acceptance of the quality control plan the Contractor shall receive the contracting officer’s acceptance in writing of any proposed change to its QC system.

C.5. PERSONNEL QUALIFICATIONS

C.5.1. KEY PERSONNEL

The following positions are considered key personnel:

Program Manager/Alternate Program Manager: The Contractor shall provide a program manager who shall be responsible for the performance of the work. The name of this person and an alternate who shall act for the Contractor when the manager is absent shall be designated in writing to the contracting officer. The program manager or alternate shall have full authority to act for the Contractor on all contract matters relating to daily operation of this contract. The program manager or alternate shall be available between 9:00 a.m. to 5:30p.m. Monday thru Friday except Federal holidays or when the government facility is closed for administrative reasons. The Program Manager must have at least 5 years relevant experience managing projects or programs like the work described in this SOW.

Crawl Engineer: The Contractor shall provide a crawl engineer who is responsible for the management of and reporting on the harvesting activities. The crawl engineer must have at least five years relevant experience managing web harvesting crawls.

Quality Assurance Lead: The Contractor shall provide a quality assurance lead who is responsible for ensuring that the QAP is followed by any Contractor staff. The Quality Assurance Lead must have at least 2 years relevant experience performing quality assurance on web archives.

C.5.2 DELETION OF CAPTURED CONTENT

At the end of each task order, upon confirmation from the Library that all copies of captured content and crawl logs and reports have been successfully transferred to the Library, the Contractor must delete all copies of captured content stored by the Contractor.

C.6. PHASE IN /PHASE OUT PERIOD.

To minimize any decreases in productivity and to prevent possible negative impacts on additional services, the Contractor must have personnel on board, during the thirty (30) day phase in/phase out periods. During the phase in period, the Contractor must become familiar with performance requirements in order to commence full performance of services on the contract start date.

(End of Performance Work Statement / Statement of Work)

030ADV26R0010 Section F

SECTION F - DELIVERIES OR PERFORMANCE

F.1 LC52.215-5 PLACE AND PERIOD OF PERFORMANCE (JAN 2019)

Place of Performance: The place of performance shall be the contractor’s own facilities.

The overall period of performance shall be as stated in section B.

The contractor is responsible for determining its hours of operation in fulfilling the requirements of this contract. The Library’s standard operating hours for this requirement are 9:30 AM through 5:30 PM eastern time, Monday through Friday, excluding Federal holidays. The contractor shall coordinate with the COR for any work to be conducted at Library facilities or any access to digital Library environments.

(End of Clause)

030ADV26R0010 Section G

SECTION G - CONTRACT ADMINISTRATION DATA

G.1 LC52.201-3 CONTRACT ADMINISTRATION (AUG 2015)

This contract will be administered by:

Library Contracting Officer: Jennifer Zwahlen

Phone: 202-707-0925

Email: jzwa@loc.gov

Library Contract Specialist: Colleen Daly

Phone: 202-707-0386

Email: cdaly@loc.gov

Library Contracting Officer Representative: TBD on award

Phone: TBD on award

Email: TBD on award

Contractor Contract Administrator: TBD on award

Phone: TBD on award

Email: TBD on award

G.2 LC52.232-1 LIBRARY OF CONGRESS INVOICE INSTRUCTIONS (DEC 2022)

The Contractor must prepare and submit invoices electronically to: https://www.ipp.gov. The Contractor may submit payment requests using other than electronic means only when alternate procedures are authorized by the contracting officer in writing. Assistance is available via the IPP Help Desk via email at: ippcustomersupport@fiscal.treasury.gov or by commercial telephone at (866) 973-3131. Invoices shall contain the information required in FAR 52.212-4, paragraph (g).

(End of Clause)

G.3 LC52.232-2 SCHEDULE OF PAYMENTS (JUN 2015)

Firm fixed price deliverable quantities will be identified at the Task Order level. The contractor must submit invoices for payment upon delivery of the successful performance of the events identified in each Task Order pursuant to the technical requirements outlined in Section C in this contract and any additional requirements included in each discrete Task Order.

030ADV26R0010 Section G

030ADV26R0010 Section H

SECTION H - SPECIAL CONTRACT REQUIREMENTS

H.1 Use of Generative Artificial Intelligence (AI)

The contractor shall not use generative AI (including, but not limited to ChatGPT, BARD, DALL-E, MidJourney, or Stable Diffusion) to create or produce any deliverables under this contract without specific written authorization from the Contracting Officer. Generative AI systems and tools are those which produce synthetic text, images, code, video, and music in response to user prompts, and include, but are not limited to, ChatGPT, BARD, DALL-E, MidJourney, and Stable Diffusion). For purposes of these requirements, generative AI does not include the use of those AI-enhanced word processing, email, or search tools that are already widely incorporated in day-to-day life such as AI tools embedded in image editing applications or code incorporated into commercial off the shelf products (COTS).

Unless specifically authorized in writing by the Contracting Officer, the Contractor shall not use Library furnished data provided under this contract to feed, train, supplement, or otherwise advance generative AI software.

H.1 LC52.201-1 CONTRACTING OFFICER'S REPRESENTATIVE (COR) APPOINTMENT AND

AUTHORITY (JAN 2016)

(a) Performance of work under this contract is subject to the technical direction of the COR. The term “technical direction” includes, without limitation, direction to the contractor that directs or redirects the labor effort, shifts the work between work areas or locations, and/or fills in details and otherwise serves to ensure that tasks outlined in the contract are accomplished satisfactorily.

(b) Technical direction must be within the scope of the contract specification(s)/work statement. The COR does not have authority to issue technical direction that:

(1) Constitutes additional work outside the contract specification(s)/work statement;

(2) Constitutes a change as defined in the “Changes” clause of this contract;

(3) Causes an increase or decrease in the contract price, or the time required for contract performance;

(4) Changes any of the terms, conditions, or specification(s)/work statement of the contract;

(5) Interferes with the contractor's right to perform under the terms and conditions of the contract; or

(6) Directs, supervises or otherwise controls the actions of the contractor's employees.

(c) Technical direction may be oral or in writing. The COR must confirm oral direction in writing within five workdays, with a copy to the Contracting Officer.

(d) The Contractor shall proceed promptly with performance resulting from the technical direction issued by the COR.

If, in the opinion of the contractor, any direction of the COR or the designated representative falls within the limitations of (b) above, the contractor shall immediately notify the Contracting Officer no later than the beginning of the next Government work day.

(e) Failure of the Contractor and the Contracting Officer to agree that technical direction is within the scope of the contract shall be subject to the terms of the “Disputes” clause of this contract.

(End of Clause)

H.2 LC52.203-1 CONTRACTOR PUBLICITY (OCT 2013)

The Contractor, or any entity or representative acting on behalf of the Contractor, may not refer to the equipment or services furnished pursuant to the provisions of this contract in any news release or commercial advertising, or in connection with any news release or commercial advertising, without first obtaining explicit written consent to do so from the Contracting Officer. Should any reference to such equipment or services appear in any news release or commercial advertising issued by or on behalf of the Contractor without the required consent, the Government will consider institution of all remedies available under applicable law.

(End of Clause)

H.3 LC52.204-1 COLLECTIONS SECURITY (DEC 2013)

(1) The Library is a public institution responsible for making its resources (collections and staff) available to the Congress and the American people. To achieve a balance between access and security, the Library requires everyone (staff, visitors, interns, contractors, etc.), to always use due diligence and protect its assets during use.

(2) Physical access to Library collections is limited to staff and/or determined by the host office. In addition, established requesting processes in the various readings rooms must be followed.

(3) Loan Privileges. Contractor employees may obtain loans of Library property for internal use and Library work-related purposes.

i. Loan of Library Property. All loans of Library property must be approved and documented by the host Library Services office. Contractor staff shall obtain a Library “General Pass” (Form LW 12/54 (rev2/88)) for each loan.

ii. Liability – Loss or Damage of Library Property. Use or loan of all Library property and signature on this contract means that the contractor acknowledges and agrees to:

(a) ensure the return all Library property issued in the same condition as borrowed;

(b) accept responsibility and liability for the negligent loss or damage of issued or borrowed Library property; and

(c) ensure that the loaned property is used for Library purposes and not loaned to any other person.

(End of Clause)

H.4 LC52.204-3 INFORMATION SYSTEMS SECURITY (JUL 2024)

(1) Computer Security Policy. The Library requires anyone using or accessing its digital assets (including networks, servers, workstations, systems, data, web pages, and email) to protect them and use them only for official business, and only in accordance with Library regulations and applicable Library bargaining agreements. Unauthorized or inappropriate use may be grounds for termination or other contractual remedy.

(2) Privacy Policy. No one has the right to privacy while using any Library computer system, including internet or email services. Usage may be monitored.

(3) Level of Access. The Library’s Host Office will determine the level of access to the Library’s IT equipment, staff and software needed to successfully perform under this contract. Access may include, for example, email, word processing, internet, and intranet.

(4) Mandatory IT Security Training. Contractor staff must successfully complete mandatory information systems security training and access forms prior to use of or access to any of the Library’s digital assets.

(a) The required training is available on the Library’s learning management system, LOC Learn, at https://www.loc.gov/extranet/cld/. Library staff will provide further instructions upon onboarding.

(b) Contractor staff with access to Library digital assets shall complete the information systems security training annually. The training is available on the Library intranet at https://www.loc.gov/staff/cld/.

(End of Clause)

H.5 LC52.204-4 CONTRACTOR EMPLOYEE FITNESS (SEP 2024)

“Contractor employee fitness” means fitness based on character and conduct for work for or on behalf of the Government as a contractor employee.

Work under this contract is unclassified. However, the Library of Congress shall conduct background investigations of contractor personnel who will have 1) regular, ongoing, unescorted access to the Library’s buildings, grounds, or collections, 2) access to Library information technology systems, and/or 3) access to Library information not available to the public, including information determined to be Controlled Unclassified Information (CUI), to ensure they meet basic standards of honesty and trustworthiness pursuant to Library regulations. The Library does not determine the fitness of employees to work for the contractor firm that hires them. The Library’s interest is in determining whether contractor personnel should be granted greater access than the general public to Library facilities, Library information technology systems, and/or non-public Library information.

(1) Any access to non-public Library facilities, Library information technology systems, and/or non-public Library information will require a contractor employee fitness determination. All initial, existing, and reciprocal contractor employee fitness determinations will be subject to continuous vetting. Continuous vetting will vary based on position risk level. An example of continuous vetting is outlined in Trusted Workforce 2.0 (https://www.performance.gov/trusted-workforce/ and https://www.dcsa.mil/Personnel-Security/Continuous- Vetting/).

(2) A favorable contractor employee fitness determination requires completion and successful adjudication of a security application request (release forms, fingerprints, and a full background investigation). To facilitate the fitness determination, the prospective contractor employee shall submit to the Library complete and accurate information as identified on the National Background Investigation Services (NBIS)’s eAPP (https://eapp.nbis.mil) or other system as provided by the Personnel Security Division (PSD). Contractors should anticipate a minimum of two weeks for the process, provided all requested information is timely, accurate and complete and that no adverse information is received. A background investigation generally will require that the candidate has cumulatively resided in the United States for at least three of the past five years. This requirement is subject to certain exceptions assessed on a case-by-case basis by PSD in accordance with established federal personnel vetting standards.

(3) After the contractor employee fitness determination, each contractor employee with a favorably adjudicated background investigation shall be enrolled in and subject to the Library’s continuous vetting process. Enrollment in the Library’s continuous vetting process also requires enrollment in the Federal Bureau of Investigation (FBI) Record of Arrest and Prosecution Background (“Rap Back”) program, which provides the Library with notification of any criminal activity by the enrollee. Upon enrollment in Rap Back, the Library shall provide each enrollee with a notification of their rights with respect to the collection and use of their fingerprints and to the accuracy of the information obtained from Rap Back, as well as a copy of the FBI Privacy Act Statement that describes how FBI may use their fingerprint images.

(4) The fact that the Library may conduct background investigations and continuous vetting on assigned contractor personnel does not relieve the contractor of the responsibility to provide qualified, reliable personnel of sound character and conduct. This includes providing contractor personnel who can maintain compliance with the Library’s requirement of a drug-free workplace and federal laws regarding controlled substances (see the Controlled Substances Act, 21 U.S.C.

§ 801 et seq. for the definition of controlled substances). In the event that drug testing is required for a contractor employee to obtain a favorable fitness determination, the contractor will be responsible for bearing the costs of such testing, conducting the testing in a manner consistent with federal vetting standards, which are based on the testing protocols published by the Substance Abuse and Mental Health Services Administration (SAMHSA), and providing the results to the Library.

(5) The Library may require removal of contractor staff from Library facilities and revocation of access to Library information…

This is the start of the file's text. The full file is on GovTribe.

File details come from the government source that posted it. Updated .