J9 - QA_030ADV26R0010_20260213 (pp. 7).pdf

Attachment J9 – Q&A 030ADV26R0010 – Web Harvesting Services – pp. 6

February 13, 2026

Questions and Answers

Q1: Attachment J2a – Sample Web Crawl states that information regarding the size of the crawl is due to the Library Contracting Team by February 2, 2026. Please confirm what date this information is due by.

A1: Information regarding the size of the sample web crawl is due to the Library Contracting Team by 5:00pm eastern time on February 17, 2026 (one week before the due date for all proposal materials). Please see the amended Attachment J2a posted with Amendment 0001.

Q2: Attachment J3 – Past Performance Questionnaire states that PPQs are due to the Contracting Team by February 9, 2026. Please confirm what date this information is due by.

A2: Past Performance Questionnaires are due by noon eastern time on February 24, 2026.

Questionnaires must be sent to the Contracting Team directly from the past performance reference who will complete the PPQ. Questionnaires sent to the Library by the offeror will not be considered.

Q3: Does the Library require or prefer to use any particularly identifying user agent for the sample crawl (Section L.4.1 Volume I, Subfactor II and Attachment J2a) with the crawl techs a business uses?

A3: The Library does not require the use of a particular user agent. Offerors may use their own user agent or a user agent of their choice. As the offeror is not crawling on the Library’s behalf for the sample test crawl (i.e. the sample will not be archived or added to Library collections), the offeror shall not use a user agent indicating the Library of Congress for this test.

Q4: In C.3.4 of the solicitation, the Contractor is required to store and maintain data in the requisite infrastructure (both primary copy and disaster recovery copy). Would the Library consider an approach where crawl data is written by the Contractor, directly into cloud infrastructure owned and controlled by the Library?

A4: No

Q5: Are you able to provide a ballpark estimated TB expected to be crawled over the course of the project, or per year? If that’s not possible, could you share how many TB have been crawled annually in the current or previous contract?

A5: We anticipate a range of 300-700 TB a year, however the actual number may vary dependent on the cost per terabyte.

Q6: For sites where robots.txt support is required, would an implementation that checks whether HTML pages are disallowed in robots.txt rules be sufficient, or would all in-page resources need to be checked as well? For example, if a site allows HTML but not stylesheets or javascript on a certain page, adhering to all robots.txt exclusions on that page will generally guarantee a bad/incomplete capture.

A6: Where crawl instructions include a respect robots.txt directive, either implementation could be acceptable depending on the specifications and allowances of the web harvest tools in use by the contractor.

Q7: For the test crawl, is the expectation that the entirety of the provided seeds will be crawled in the same 48 hour period, or over a 48 hour period? For example, if one seed is crawled between 12:00pm Monday and 12:00pm Wednesday, does that imply all other seeds must also crawled within the same 48 hour period, or can other seeds be crawled in a different 48 hour period, for example, between 1:00pm Monday and 1:00 Wednesday?

A7: The sample web crawl should include the full list of sample seeds in one crawl lasting no longer than 48 hours, within the same 48-hour period.

Q8: For the current/previous contract, we see that “[t]he.calculated.total.evaluated.price.

proposed.by.MirrorWeb.is.₰996?666¡66‹ .in the award notice for the current five-year contract publication (https://sam.gov/workspace/contract/opp/0c421d7528fe4878895c20a61b3200b3/view) - does that price correspond to the total 1500 TB estimated to be crawled across the full five year project in the “Attachment J4 - Evaluation Pricing Schedule.xlsx” from that corresponding request for proposals (https://sam.gov/workspace/contract/opp/ac1121196c62452c95020c8a34093a4c/view)? If the actual price of the previous/current contract was different than the initially proposed price, could the Library share the total price of that contract?

A8: This is a new solicitation for a new and completely separate contract. This question pertains to the previously awarded contract and it is not relevant to this solicitation.

The Library does not provide pricing information for existing contracts.

https://sam.gov/workspace/contract/opp/0c421d7528fe4878895c20a61b3200b3/view https://sam.gov/workspace/contract/opp/ac1121196c62452c95020c8a34093a4c/view

Q9: If it is necessary to take a crawl as scoped and defined by the Library and split it into a number of smaller crawls (e.g. per-seed, as mentioned in 3.1.10.2) for best results, would the Library accept reports per actual crawl run, or would the reports need to be aggregated to match the crawls as defined by the Library in task orders?

A9: Either option could be acceptable, as long as requirements in C.3.3 Reports are met.

Q10: Following on from the previous RFI (https://sam.gov/workspace/contract/opp/2048303e99574b2ea00b0887cb78072f/view), the Library asked “Is.it.realistic.for.a.single.company.to.do.both.high‗volume.crawls.and.high‗ fidelity.small‗scale.crawls.in.the.scope.of.this.contract‽” Based on the feedback to the RFI, would the Library still consider the possibility of awarding a separate contract for high fidelity browser-based crawling during the five-year period covered by this project? Or is it imagined that this project will encompass all contracted crawling services for the Library during the coming five year period?

A10: This is a new solicitation which will result in a new contract. The RFI has closed. The RFI is not relevant to this solicitation. The opportunity is already described in the solicitation.

Q11: Is the Library no longer crawling social media posts, and especially those behind logins, as a matter of policy or because it hasn’t been technically feasible? Would the Library be interested in this capability if it were available?

A11: See Section C.1. Overview/Background

Q12: In C.1 Background / Overview, the Library states that roughly 70% of 16,500 seeds are crawled with Heritrix and the remaining 30% are crawled using browser-based crawling. Can the Library share which tools are currently used for browser-based crawling? If the 30/70 breakdown is for the total number of seeds, is a volume of data crawled percentage breakdown available, eg. what percentage of total crawl volume (in TB) the Library obtains through browser-based crawling vs Heritrix/large scale crawling.

A12: Browser-based crawling tool usage and volume varies by vendor. No further breakdown is available.

Q13: In Attachment J.1, it stated the “The.Library"s.web.harvesting.curator.tools.and.

infrastructure.that.support.the.digital.lifecycle.for.web.archives.were.initially.developed.for.

the.inputs.and.outputs.of.Heritrix.and.later.Brozzler.for.harvesting?.and.OpenWayback.

(currently.version.8¡0¡6)?.and.later.pywb.(currently.version.8¡❹¡0).and.OutbackCDX.(7¡6¡6).for.

access¡.These.software.tools.are.open.source.and.are.supported.by.the.International.Internet.

Preservation.Consortium?.of.which.the.Library.is.a.founding.member.(netpreserve¡org)¡” https://sam.gov/workspace/contract/opp/2048303e99574b2ea00b0887cb78072f/view) http://netpreserve.org/

Does the Library see a direct role in supporting the development and maintenance of key tooling that its infrastructure depends on, either within this contract or further proposals, rather than relying on whatever support may or may not be available through external entities such as the IIPC? The price schedule suggests work orders are to be priced around TBs captured, but would the Library consider additional work items related to development, maintenance and upkeep of open source tooling that the Library relies on as part of its web archiving program?

A13: The opportunity is already described in the solicitation. The Library is not contemplating additional contracts.

Q14: Considering the complexity of the solicitation and the level of technical detail required, is the Government considering an extension or postponement of the proposal submission deadline to 15 March?

A14: The Library has not suggested any extension of the proposal submission deadline. Proposals are due per the solicitation closing date.

Q15: Beyond Heritrix references, does the Government have any preferred or restricted crawler technologies, particularly for browser-based crawling?

A15: No

Q16: Is there a defined retention period for crawl data in the Contractor’s environment prior to transfer and acceptance by the Library?

A16: See solicitation section C.3.4.3 Duration of Content Storage

Q17: Instead of PPQ's can the government accept CPARS?

A17: The past performances requirements are already described in the solicitation.

Q18: Can the Library further define what constitutes “highly complex sites” (e.g., SPA frameworks, authenticated workflows, heavy JavaScript rendering) for planning crawler strategies?

A18: No

Q19: For mid-crawl adjustments initiated by the contractor (Section 3.1.12), is Library approval required before resuming, or is notification within 24 hours sufficient?

A19: The contractor would notify and take action, such as resuming an interrupted crawl, to resolve the interruption of service (C.3.1.15). Library approval is not required to resume normal operations.

Q20: Are there performance benchmarks (page load times, concurrency thresholds, availability targets) for the contractor-provided access tool beyond availability windows?

A20: Access tool requirements are already described in solicitation section C.3.2.

Q21: For cloud-based services, does “servers physically located in the United States” allow use of multi-region U.S. availability zones, or must crawling be constrained to a single region?

A21: Yes, “servers physically located in the United States” C.3.1.17 could include multi-region U.S.

availability zones.

Q22: Does the Library of Congress have any preferred experience profiles, role types, or minimum experience thresholds for key personnel supporting this IDIQ (e.g., years of relevant experience, prior web archiving programs, or federal records/preservation experience), or should offerors propose key personnel based solely on their judgment of what best supports successful performance?

A22: The solicitation contains the requirements for key personnel in section C.5.1.

Q23: Beyond the evaluation factors identified in Section M, does the Library of Congress have any preference regarding the relative emphasis within the technical volume (e.g., approach/methodology, management processes, and quality assurance controls)?

A23: The evaluation will be conducted as described in Section M.

Q24: Does the Library of Congress prefer past performance to be presented exclusively as a standalone volume, or may offerors also reference and integrate relevant past performance examples within the technical and/or management volumes?

A24: The instructions for submission of past performance are already included in solicitation section L.

Q25: Where the solicitation is silent or ambiguous, is the Library of Congress receptive to clearly stated assumptions and constraints within proposal narratives, or should offerors avoid explicitly stating assumptions and focus solely on the requirements as written?

A25: The Library cannot advise offerors on the best approach to writing their proposal. Offerors that choose to submit proposals shall respond to the best of their abilities per the solicitation requirements.

Q26: The current language prohibits the use of "Generative AI" to produce deliverables without written authorization. Does the Library distinguish between "Generative AI" (synthetic content creation) and "Analytical" or "Agentic AI" used for autonomous orchestration, quality assurance validation, and metadata extraction (e.g., DocAI for PDF processing)?

A26: Not at this time. However the field is emerging and policy may evolve over the coming years.

Q27: Can the Library provide the technical schema or API specifications for Digiboard? This will ensure that automated routing agents can ingest SURT-formatted seed lists without manual intervention.

A27: This is not possible

Q28: Section C.3.1.11 encourages testing new tools for "challenging content." Will the Library consider "Agentic" browser-based crawling, which simulates human-like interaction to bypass log-in walls or trigger lazy loading as an approved approach under Section C.3.1.7?

A28: Refer to H.1 “Use of Generative Artificial Intelligence.”

Q29: Who is the incumbent?

A29: The Library does not provide this type of information.

Q30: The Library evaluates price per terabyte. For pricing purposes, will the "Total Compressed WARC size" or the "Total Raw Data size" be the primary metric used to calculate the invoiced volume?

A30: The total compressed WARC size will be used to calculate the invoiced volume.

Q31: Please could we request a short extension to the February 17th deadline to send the sample crawls?

A31: To clarify, by Feb 17 we need this information: "size, or estimated size, of the sample crawl data and reports" via email. (this is in J2a - Sample Web Crawl Requirement)

Q32: For the SFTP transfers, please could you also share the specific details for the transfer: Username, Password, Port and URL. We are unable to send without these details.

A32: Once we have the above data about size, we will be in touch with further instructions for how to transfer the data to the Library for review by the 5:00 PM EST on 02/24/2026 (the proposal deadline.) The transfers must be complete before this deadline.