Attachment 1- Statement of Work.pdf

ATTACHMENT 1-STATEMENT OF WORK (SOW):

Project "NALT for the Machine Age" (Annif)

This project will provide continuing support for the NAL Thesaurus (NALT) for the Machine Age Initiative (N4MA) onto a new platform and support to be provided by USDA ARS Partners in Data Innovations (PDI), that will allow wider accessibility for ARS staff. Also, this project will provide training to optimize the expertise of the new NALT Concept Space central curation team, with consultation to facilitate the design and maintenance of sub-schemes within NALT by external communities of experts to meet their unique needs. This expert support, consultation, and training will allow NAL to leverage machine learning and artificial intelligence, both for automated subject indexing using NALT sub-schemes, and to support the curation and visualization of NALT Concept Space itself. Initiation of editorial user groups and best practices will be developed, documented, and coordinated, and expert consultation provided via design, development, and support for developing human and automated workflows and their integration as appropriate, to process NALT Concept Space through the VocBench curation software, and for publishing NALT in SKOSMOS or other appropriate online display.

The editorial and curation teams will be established and the ongoing N4MA work will continue as NALT evolves both in terms of technical advances and optimization, management, and content. The goal is for NALT to be state-of-the-art now, and to create the computer and human workflows that will maintain this designation in perpetuity as the technology advances. The N4MA has also expanded to partner with the PDI to use NALT concepts for developing data shapes using ShEx (shapes expression). This contract will generally support, through expert opinion/feedback as needed, the integration of data shapes into the overall workflow and usage of NALT.

The following work will run over the period of performance 9/30/2023 – 9/29/2028 (base plus four option years).

Automated subject indexing testbed and evaluation

The contractor shall provide support and consultation for a testbed for subject indexing as a sandbox for training and development, and as soon as it is available, to be deployed for production on the USDA PDI platform. This will be done by training the open-source platform Annif, from the National Library of Finland (NLF), with NALT and NALT Core, and its results will be compared with the results of existing indexing systems. Following on to this work NAL staff will be trained to use, maintain, and support the testbed, and assist NAL patrons partnering with the NALT vocabulary.

This SOW will focus on building pipelines for a workflow connecting functionality from vocabulary building, displaying, and archiving (from VocBench to SKOSMOS), and to training NAL staff to utilize SKOS files with NAL or other data to build models with Annif, and make them accessible to users.

The Annif Project at the National Library of Finland has developed an open-source platform for automating the process of subject indexing. When trained using subject vocabularies such as NALT, Annif can analyze the contents of scientific papers using a combination, or "ensemble", of open-source natural language processing and machine learning algorithms and propose subject headings for approval by expert indexers. The consultant will work with the Indexing and Informatics Branch (IIB) of NAL, ideally in collaboration with NLF, on implementing an Annif testbed at NAL.

1. Dump Extract data sets from NAL systems. Follow on to completed work in prior contract as needed and per consultant recommendations. The data sets used for training and evaluating Annif with NALT – for example, AGRICOLA, PubAg or research community of practice data, and the Ag Data Commons data catalog - must be extracted from relevant NAL systems. The result should be dumps of each data set, with essential document metadata (title, abstract, date of publication, assigned NALT subjects) in an easily understood format (e.g. CSV, XML, JSON).

a. Who: NAL, with help from consultants

b. Timeframe: December 2024, with ongoing support throughout POP and in each option year

c. Deliverables: Database dumps

d. Dependencies: NAL provides data timely.

2. Corpus Training or support for NAL staff to use data conversion tools (to Annif corpus format), train Annif models, and perform numeric evaluations. Train NAL staff to use tools created in prior contract, to convert into a corpus format understood by Annif, including splitting up the data sets into training, validation, and evaluation subsets as necessary, primarily by date of publication. Train NAL staff in using Annif on the USDA platform – upload Vocabularies, configure, build, and evaluate models, and display on the USDA platform, and port models to the NAL Annif instance for public access.

a. Who: NAL, with help from consultants

b. Timeframe: March 2024, with ongoing support throughout POP and in each option year

c. Deliverables: Data sets in Annif corpus format; Conversion scripts with documentation, Evaluation result report; Choice of algorithms and their configurations; Final trained Annif models on the USDA platform Annif, and portable to NAL Annif.

d. Dependencies: Follow on to completed work in prior contract for

1. Dump Extract data sets from NAL systems

e. Dependencies: NAL data, NAL staff, NAL and USDA Platform Annif instances, IT resources and support provided timely.

3. Pipeline VocBench to SKOS outputs, search, files, as proof of concept, etc. The workflow will be built as a proof-of-concept sandbox, to inform deployment for production on a USDA platform. In the interim hybrid system is proposed, whereby

VocBench created vocabulary slices would be used to build Annif models on the sandbox or USDA platform. These models and vocabulary slices would ultimately be deployed on a USDA platform for public access. For development/management: the system should provide a fully functional system, with NAL and the USDA platform support to use, maintain, and support it. VocBench user support final deliverables may be adjusted based on contractor recommendations.

a. Who: Contractor

b. Timeframe: Ongoing in each option year through each POP.

c. Deliverables:

i. GitHub, archiving…

ii. Versioning…

iii. Format conversions

iv. Quality Control

v. Downloadable files

vi. SKOSMOS data

vii. VocBench User Groups AI Support–

1. SKOS slices for SKOSMOS presentation, and

2. Annif modeling by NAL team (as NAL provided service or support to user groups)

3. Annif access to their group’s Annif models and vocabularies (NALT slice)

d. Dependencies: The USDA platform support and infrastructure, IIB data scientist staff, NAL Annif instance support.

4. Annif Generate test annotations with Annif. The test documents need to be annotated using Annif to get the NALT subjects which would be suggested if Annif were used instead of Cogito or other indexing resources.

a. Who: NAL, with contractor support

b. Timeframe: by December 2023-April 2024, with ongoing support throughout POP and in each option year

c. Deliverables: Set of test documents with Annif-suggested NALT subjects

d. Dependencies: T2.docs Extract documents for tool comparison; T2.eval Numeric evaluation and fine-tuning of algorithms

5. Compare non-Annif (i.e., Cogito, others available, and golden sets as a standard) annotations with Annif annotations. The annotations from other indexing production or quality control efforts and annotations from Annif should be compared to find out which have higher quality. Contractor will set up a google spreadsheet or similar tool to track the comparison. This would likely involve NAL indexers (or other NAL or USDA subject specialists) rating the suggested subjects for the test documents in a workshop setting, without knowing whether they were produced by Annif or other indexing production streams such as Cogito.

a. Who: NAL, with contractor support

b. Timeframe: by January-May 2024, with ongoing support throughout POP and in each option year

c. Deliverables: Comparison results report

d. Dependencies: NAL to generate test annotations with Cogito, or other indexing resource; #4. Annif Generate test annotations with Annif, Golden set annotation standards for calibration/measuring Annif annotations.