STATEMENT OF WORK-NALT for the Machine Age.pdf

STATEMENT OF WORK

Project "NALT for the Machine Age"

This project will upgrade the NAL Thesaurus (NALT) for the Machine Age and optimize the expertise of its central curation team. This will be accomplished by facilitating the maintenance of sub-schemes within NALT by external communities of experts to meet their unique needs, and through leveraging Machine Learning and Artificial Intelligence, both to automate subject indexing using NALT and to support the curation of NALT itself.

The following three work packages will run concurrently from June 2020 through August

2021.

WP1: NALT structure and workflows

Work Package I evaluates the structure of NALT and existing workflows for its curation.

It assesses the needs of NALT users with particular attention to the requirements of applications driven by AI and machine learning. Phase I also explores how external communities of experts might help maintain specific subsets of NALT. The resulting roadmap of next steps will be discussed and revised according to feedback from project participants.

T1.snap Extract NALT snapshot. The most recent published NALT version, from

January 2020, is already a bit dated. NAL will extract a more recent snapshot from the MultiTes vocabulary management system and convert it to SKOS format so that it can be used by the consultants.

o Who: NAL o Timeframe: June and July 2020 o Deliverables: NALT SKOS file o Dependencies: None

T1.core Create NALT Core. The consultant will create NALT Core as new concept scheme composed of the 12,000 NALT concepts (out of 74,000) that are mapped to GACS Core.

o Who: Tom o Timeframe: June and July 2020 o Deliverables: NALT Core concept scheme o Dependencies: T1.snap Extract NALT snapshot

STATEMENT OF WORK: Tom Baker, and Osuuskunta Sange (Osma Suominen)

T1.gacs Integrate GACS concepts not already mapped to NALT. Some 3,000

GACS Core concepts have no equivalent in NALT. The consultant will list these concepts in a Google sheet for review by IIB staff. A decision will be made whether to integrate these as new concepts into NALT Core.

o Who: Tom, possibly also IIB staff o Timeframe: June and July 2020 o Deliverables: List and analysis of GACS concepts that do not yet exist in

NALT

o Dependencies: T1.snap Extract NALT snapshot, T1.core Create NALT Core

T1.struct Structural analysis of NALT. The consultant will analyze the structural features of NALT with particular attention to changes made since 2014 and to differences between NALT Full and NALT Core.

o Who: Tom o Timeframe: August 2020 o Deliverables: Report on the structure of NALT o Dependencies: T1.core Create NALT Core

T1.workflow Analysis of maintenance workflow. The consultant will explore the processes, criteria, workflows, and editorial software (including NALT

Annotator) by which the IIB curation team integrates new terms into NALT and analyze the current practices for their strengths and weaknesses through interviews with members of the IIB Team.

o Who: Tom, IIB Team o Timeframe: August to November 2020 o Deliverables: Interview schedule (IIB Team), Report on NALT maintenance workflows (Tom) o Dependencies: NAL list of people to interview

T1.plat Selecting and deploying a maintenance platform. The consultant will help IIB to evaluate one or two alternative SKOS-native editorial platforms in light of requirements such as support for multiple sub-schemes; concurrent editing;

usability by external subject specialists; and software maintainability by the NAL

IT department.

o Who: NAL with help from Tom and Osma, NAL IT department o Timeframe: November to December 2020 o Deliverables: NALT maintenance platform such as Vocbench, Pool Party, or

TopBraid o Dependencies: T1.core Create NALT Core

T1.load Loading NALT full into the SKOS maintenance platform. The consultant will load NALT Core into the maintenance platform.

o Who: Tom o Timeframe: November to December 2020 o Deliverables: New NALT maintenance platform with NALT Full loaded o Dependencies: T1.plat Selecting and deploying a maintenance platform

T1.sub Setting up NALT Core as a sub-scheme within NALT Full. The consultant will load NALT Core into the maintenance platform as a distinct sub-scheme within NALT Full.

o Who: Tom o Timeframe: December 2020 o Deliverables: NALT maintenance platform such as Vocbench, Pool Party, or

TopBraid o Dependencies: T1.core Create NALT Core

T1.req NALT user requirements. The consultant will interview circa twenty NAL staff and partners, including staff indexers, data scientists, product and service maintainers, and external researchers working with AI applications in DPD, KSD, IPD, ARS, the NAL Branch for Artificial Intelligence, SCINet, and in various projects both external and in-house, about existing annotation processes, the use of NALT for in-house projects, and the use of NALT, both existing and potential, in AI-driven products and services. The analysis will highlight features of NALT required by AI- and ML-driven applications.

o Who: Tom in consultation with IIB and organizations using NALT o Timeframe: December 2020 to February 2021 o Deliverables: Report on NALT features required by users (Tom) o Dependencies: NAL list of people to interview

T1.ext External curation processes. The consultant will propose processes for engaging and integrating expert communities into the ongoing curation, development, and evaluation of NALT, taking into account the potential for collaboration with maintainers of related and complementary concept schemes.

o Who: Tom in consultation with IIB Team and potential curation partners o Timeframe: February 2021 o Deliverables: Draft guidelines for IIB Team and curation partners o Dependencies: T1.req NAL user requirements

T1.guide NALT curation guidelines. The NALT curation guidelines will be revised for use not only by the IIB team but also by external curation partners on the basis of existing guidelines for NALT, GACS, and AGROVOC.

o Who: Tom in consultation with IIB Team and cooperation partners such as

FAO

o Timeframe: June to November 2020 o Deliverables: Documentation website and PDF of revised curation guidelines o Dependencies: T1.struct Structural analysis of NALT

T1.animal External curation testbed: Animal Welfare. The consultant will help implement and test the external curation processes on the example of the NAL in-house project on Animal Welfare. After two or three months, the consultant will review and evaluate the experience and summarize any lessons learned in a report.

o Who: NAL, with help from Tom o Timeframe: February to June 2021 o Deliverables: Evaluation of external curation processes (Animal Welfare) o Dependencies: T1.ext External curation processes

T1.dataset External curation testbed: Dataset variables. NALT has traditionally been used for indexing bibliographic resources but could potentially provide authority control for variables found in field observations or experimental data.

This task could explore the potential for including such variables into the scope of

NALT if an appropriate curation partner could be found.

o Who: NAL, with help from Tom o Timeframe: February to June 2021 o Deliverables: Evaluation of external curation processes (Animal Welfare) o Dependencies: T1.ext External curation processes, NALT curation partners

WP2: Automated subject indexing testbed and evaluation

A testbed for subject indexing will be created by training the open-source platform

Annif, from the National Library of Finland (NLF), with NALT and NALT Core, and its results will be compared with the results of existing indexing systems.

The Annif Project at the National Library of Finland has developed an open-source platform for automating the process of subject indexing. When trained using subject vocabularies such as NALT, Annif can analyze the contents of scientific papers using a combination, or "ensemble", of open-source natural language processing and machine learning algorithms and propose subject headings for approval by expert indexers. The consultant will work with IIB, ideally in collaboration with NLF, on implementing an Annif testbed at NAL.

T2.dump Extract data sets from NAL systems. The data sets used for training and evaluating Annif with NALT - tentatively AGRICOLA, PubAg and the Ag Data

Commons data catalog - must be extracted from relevant NAL systems. The result should be dumps of each data set, with essential document metadata (title, abstract, date of publication, assigned NALT subjects) in an easily understood format (e.g. CSV, XML, JSON).

o Who: NAL, with help from consultants o Timeframe: June-August 2020 o Deliverables: Database dumps o Dependencies: None

T2.corpus Convert data sets to Annif corpus format. The extracted data sets need to be converted into a corpus format understood by Annif. To do this, the consultant creates custom conversion tools as either Python CLI tools or Jupyter notebooks that perform the conversion. The conversion also includes splitting up the data sets into training, validation and evaluation subsets as necessary, primarily by date of publication.

o Who: Osma o Timeframe: July-September 2020 o Deliverables: Data sets in Annif corpus format; Conversion scripts with documentation o Dependencies: T1.snap Extract NALT snapshot; T2.dump Extract data sets from NAL systems

T2.setup Set up Annif testbed instance. To get started experimenting with

Annif, the tool needs to be installed locally on NAL systems (e.g. an Ubuntu Linux virtual machine) and configured correctly. This also includes installing some optional features of Annif (e.g. the TensorFlow based neural ensemble) and the external Maui Server tool.

o Who: NAL, with help from Osma o Timeframe: June and July 2020 o Deliverables: Annif with the necessary features running locally on one or more systems at NAL o Dependencies: None

T2.train Train Annif models. With the training data sets available in the Annif corpus format, Annif projects can now be configured and trained. Several algorithms should be used (e.g. fastText, Omikuji variants, Maui). It is expected that the initial configurations will not be optimal, but they will then be refined further during the evaluation task.

o Who: NAL, with help from Osma o Timeframe: August-October 2020?

o Deliverables: Trained Annif models in their initial configuration o Dependencies: T2.setup Set up Annif testbed instance; T2.corpus Convert data sets to Annif corpus format

T2.eval Numeric evaluation and fine-tuning of algorithms. The trained models will be evaluated using the evaluation subsets of the selected corpora.

The evaluation results will guide the final selection of algorithms and their configuration.

o Who: Osma first, then repeated at NAL o Timeframe: September-November 2020?

o Deliverables: Evaluation result report; Choice of algorithms and their configurations; Final trained Annif models o Dependencies: T2.train Train Annif models

T2.docs Extract documents for tool comparison. To be able to directly compare Annif with Cogito, a separate test set of 50-200 documents must be selected and extracted from NAL systems. The documents should be recent or otherwise unseen, so that they did not exist in the training and validation sets used for setting up Annif.

o Who: NAL, with help from Osma o Timeframe: November 2020-January 2021?

o Deliverables: Set of test documents in COGX source format o Dependencies: None

T2.cogito Generate test annotations with Cogito. The test documents need to be annotated using Cogito to get the NALT subjects which would be suggested by the existing tool.

o Who: NAL, with help from Osma o Timeframe: December 2020-January 2021?

o Deliverables: Set of test documents in COGX annotated format o Dependencies: T2.docs Extract documents for tool comparison

T2.annif Generate test annotations with Annif. The test documents need to be annotated using Annif to get the NALT subjects which would be suggested if

Annif were used instead of Cogito.

o Who: NAL, with help from Osma o Timeframe: December 2020-January 2021?

o Deliverables: Set of test documents with Annif-suggested NALT subjects o Dependencies: T2.docs Extract documents for tool comparison; T2.eval

Numeric evaluation and fine-tuning of algorithms

T2.compare Compare Cogito annotations with Annif annotations. The annotations from Cogito and Annif should be compared to find out which have higher quality. This would likely involve NAL indexers (or other NAL subject specialists) rating the suggested subjects for the test documents in a workshop setting, without knowing whether they were produced by Annif or Cogito.

o Who: NAL, with help from Osma o Timeframe: January-February 2021?

o Deliverables: Comparison results report o Dependencies: T2.cogito Generate test annotations with Cogito; T2.annif

Generate test annotations with Annif

WP3: Project communication and coordination

T3.roadmap NALT Roadmap with scenarios. The consultant will summarize the results of previous work items -- notably workflow, user requirements, external curation processes, the Annif testbed, and in a Roadmap for discussion and feedback from the IIB Team. These will be presented in a half-day workshop where members of the IIB Team and possibly other stakeholders will present work, including the Annif testbed. This could result in course corrections before implementation of the external curation testbeds.

o Who: Tom, IIB Team and other relevant stakeholders, Osma o Timeframe: February 2021 o Deliverables: Half-day workshop to discuss progress to date and roadmap scenarios o Dependencies: All preceding work items

T3.review Ongoing review of deliverables. The consultants will continually review deliverables (reports) resulting from various work items.

o Who: Consultants o Timeframe: Duration of project o Deliverables: Reviews of deliverables o Dependencies: All preceding work items

T3.comm Project communication. The consultants will meet with the Project

Lead in weekly or bi-weekly meetings, as required, and communicate on an ongoing basis via Slack and email. Progress on deliverables may be tracked using

Github Project Kanban.

o Who: Consultants, Project Lead, NAL IT (for Slack instance) o Timeframe: Duration of project o Deliverables: "Teams" meetings, o Dependencies: All preceding work items

T3.paper Journal article. The project results will be summarized in an article to be submitted for publication in a journal or presentation at a conference (or both).

o Who: Tom (lead), Osma, Jennifer, IIB Team o Timeframe: June-August 2021 o Deliverables: Draft journal article or conference paper o Dependencies: The entire project