Attachment 5 - PWS.docx

Performance Work Statement Enterprise Support for eXist February 9, 2021

Introduction

The Foreign Service Institute (FSI) is the training bureau of the U.S. Department of State (DOS) and a principal training institution for the U.S. Government foreign affairs community. The Foreign Service Institute is the U.S. Government’s premier foreign affairs training provider, serving the U.S. Department of State and the entire USG foreign affairs community as a strategic enabler of diplomatic excellence. We are dedicated to ensuring the career-long learning opportunities required for success in today’s global arena. The Institute is located at the George P. Shultz National Foreign Affairs Training Center (NFATC) in Arlington, Virginia, with additional training centers domestically, overseas, and in the Washington metropolitan area.

Purpose

The government has a requirement for enterprise-level support and development services to improve the functionality and performance of the Office of the Historian’s (FSI/OH) public website, history.state.gov (HSG), and the existing content management system, cms.history.state.gov (CMS). These are FedRAMP-low services hosted in Bureau of Information Resource Management’s (IRM) Cloud Shared Services (CSS) Amazon Web Services (AWS) commercial enclave and administered by the Office of the Historian, in compliance with DOS’s Cloud Computing Governance Board (CCGB).

Scope

FSI requires a contractor to provide all computing and support services required to architect and administer software and servers to maintain and improve FSI/OH’s public website, internal applications, and cloud-based server infrastructure.

The contractor will provide ongoing services required to operate, maintain, and safeguard FSI/OH’s existing cloud-based servers and infrastructure and develop solutions for enhancing FSI/OH’s eXist-based applications (including, as needed, improvements to the open source eXist database software package or other components), and train FSI/OH staff on the use of the applications and infrastructure.

Period and Location of Performance

The base Period of Performance is for one (1) year from the date of contract award with four (4) one-year option periods.

All phases of this effort may be performed offsite.

FSI/OH location:

U.S. Department of State, Potomac Annex (SA-4D) 2300 E Street NW Washington, DC 20372

The contractor shall be responsible for having the appropriate workspace, equipment, and material necessary to provide all deliverables and perform all duties required by the government in this PWS. The government will not provide needed equipment for remote network. Travel in and/or around the local Washington, D.C., metro area for the purpose of working at the contractor’s place of performance will not be charged to the government.

Background

The Office of the Historian (FSI/OH), within the Foreign Service Institute (FSI) at the U.S. Department of State (DOS), is responsible, under law, for the preparation and publication of the official documentary history of U.S. foreign policy in the Foreign Relations of the United States (FRUS) series. FSI/OH also prepares policy-supportive historical studies for Department principals and other agencies and conducts an array of other initiatives, ranging from briefing memos to multi-year research projects.

FSI/OH maintains a public website at history.state.gov (hereafter, HSG). In addition to a full text archive of the FRUS series, which contains over 300,000 documents from 550 books published since 1861, HSG contains a history of U.S. diplomatic relations with every state in the world, a database of all of DOS principal officers and chiefs of mission since 1789, and other numerous scholarly resources for the study of U.S. foreign relations and DOS institutional history. In keeping with the Open Government Directive’s principle of transparency, participation, and collaboration as the cornerstone of an open government, all of FSI/OH’s publications can be freely downloaded in the Text Encoding Initiative (TEI) format, an XML-based standard used widely in scholarly electronic text projects. Besides HSG, FSI/OH posts the HSG source code and raw data to the HistoryAtState organization on GitHub.

Since its launch in 2009, HSG has been built on a free, open source software package, called eXist. eXist excels in the storage and querying of XML data and has enabled the Office to contribute to all key open government initiatives on behalf of the Department, including the Open Government Directive, the Cloud First Initiative, and the Digital Government Strategy. FSI/OH has also used eXist for numerous internal applications, including a content management system (CMS). Staff install eXist and a specific suite of software on their workstations to facilitate data analysis, reporting, and development of new applications, publications, and datasets. Because eXist plays such a critical role in the Office’s internal and public-facing digital initiatives, knowledge management activities, and publishing and reporting workflows, FSI/OH requires enterprise level support and development services for eXist, applications built on it, and the server infrastructure housing it.

Current Server Environment

FSI/OH’s public website (HSG), internal application server (CMS), and various supporting servers are hosted in the DOS IRM CCS commercial Amazon Web Services enclave. These servers rely on Amazon EC2 for compute, ELB for load balancing, and S3 for storage; and are located in Northern Virginia, spread across two availability zones. The EC2 servers are running Amazon Linux 2, with the exception of one CentOS server. Software supporting these applications on the EC2 servers includes:

1. eXist (the core back-end application server and database)

2. ActiveMQ (messaging server for database replication)

3. Nginx (front-end web server)

4. Cantaloupe (image server)

5. Ansible (orchestration)

6. Jenkins (continuous integration)

7. Icinga, Munin, and Cloudwatch (monitoring)

8. OpenVPN and Unbound DNS (for secure access to the VPC)

Integrations with external systems beside Amazon Web Services include:

1. GitHub.com and GitLab.com (version control)

2. Microsoft Azure (DOS-administered single-sign on access to internal web-based applications)

3. NPM, Bower, and Maven (package management)

The EC2 servers are allocated as follows:

(Read more information about these instance types.)

Each HSG eXist-based backend database server is proxied by a frontend Nginx-based web proxy server, with traffic distributed by Amazon’s Elastic Load Balancer. Data is replicated between the two databases with ApacheMQ, using the Messaging and Replication library for eXist.

All software deployments and upgrades on the FSI/OH servers are orchestrated via Ansible. Automated web tests are performed on a per-commit or nightly basis via Jenkins on the dos-ci server.

The source code for HSG’s application logic is stored in the HistoryAtState organization on GitHub in the hsg-shell repository, and the underlying publications and datasets (all unclassified, no PII) are stored in 16 additional HistoryAtState organization’s repositories on GitHub:

1SBU - CONTRACTING AND ACQUISITIONS

1. administrative-timeline

2. carousel

3. conferences

4. frus

5. frus-history

6. gsh

7. hac

8. milestones

9. other-publications

10. pocom

11. rdcr

12. tags

13. travels

14. twitter

15. visits

16. wwdai

Required eXist packages not included in the stock eXist distribution are fetched from a public-facing eXist-based package repository (public-repo) running on the dos-ci server:

1. crypto (An implementation of the EXPath Crypto specification)

2. exsaml (for SAML support for authentication with Microsoft Azure SSO)

3. memsort (for pre-sorting large datasets; will be discontinued after adopting eXist 5’s facets and fields facility)

4. messaging-replication (for replicating eXist between a primary and replica server using ApacheMQ)

5. s3 (for interacting with the Amazon S3 API)

6. tei-publisher and lib (the TEI Publisher application and associated library for transforming TEI XML into HTML and other output formats)

7. tei-simple (a legacy version of TEI Publisher in use on hsg-shell; will eventually be discontinued after hsg-shell is migrated to TEI Publisher)

Of these, TEI Publisher is an especially important part of the application stack. In 2016, HSG migrated from custom XQuery/XSLT for the display of its TEI publications to the open source, eXist-based TEI Publisher library used widely in the TEI community. This library is responsible for the transformation of all of OH’s TEI XML data into HTML, EPUB, and PDF. (At the same time, HSG adopted the U.S. Web Design System to achieve its modern, mobile-friendly, and accessible front-end design.)

New and updated data is published to the HSG eXist servers using either (1) XML-RPC or WebDAV upload of individual files from FSI/OH workstations or (2) Upload of full application/data packages via the eXist Package Manager. The latter method currently requires the replication trigger to be manually disabled and re-enabled when packages are published, but a facility to automate the disabling/re-enabling of the replication trigger is being developed.

A second public-facing server, labs.history.state.gov, has recently been established to house seven draft-stage or experimental scholarly resources, stored in the following HistoryAtState git repositories on GitHub.com or alternatively on GitLab.com:

1. consular-cards

2. frus-indexes

3. frus-sources

4. frus-unbound

5. people

6. terms

The CMS server houses three internal applications: History Mailbox Archive (HMA), Historical Studies Archive (HSA), and Guides. The application logic and underlying data is stored in seven private HistoryAtState Git repositories on GitLab.com:

1. hma

2. hma-data

3. hsa

4. hsa-binaries

5. hsa-data

6. guides

7. guides-data

The CMS server allows FSI/OH users authorized via DOS’s Azure-based SSO facility to read and edit data. All edits to TEI XML data are synced to HistoryAtState Git repositories on GitLab.com, and all binary files are synced to a private Amazon S3 bucket. When bulk updates are performed outside the CMS server, they can be deployed to the CMS server by uploading full application/data packages via the eXist Package Manager, accessible only via VPN. As with HSG, package deployment requires manually disabling/re-enabling the replication trigger.

Current Desktop Environment

FSI/OH’s standard desktop environment for members of the digital initiatives team is called “hsg-project.” These workstations are connected to a dedicated internet network (DIN), which can access the public internet, from which they obtain the required software for the hsg-project environment—software which can also be installed on home computers used for telework. The version numbers of the software listed is current as of early 2021, but FSI/OH periodically updates to new stable versions of each.

1. macOS 10.16 (Big Sur)

2. oXygen XML Editor 23.0

3. eXist-db 5.2.0

4. Transmit 5.7.1

5. GitHub Desktop 2.6.3

6. OpenVPN Connect 3.2.5

7. Homebrew 3.0.0

8. Java OpenJDK 1.8

9. Apache Ant 1.10.8

10. Apache Maven 3.6.3

11. NodeJS 15.8.0

12. Gulp 2.3.0

13. Bower 1.8.8

After installing this software, users clone the hsg-project repository from the HistoryAtState GitHub organization, which contains and running included scripts to clone all data and source code repositories and fetch all require packages, users are able to interact with a complete local copy of the HSG website and perform all content editing, data analysis, and publishing tasks. The hsg-project repository also contains a key file, hsg-project.xpr, which is a “project file” for oXygen XML Editor that configures oXygen with schema associations, transformation scenarios, external tools for system setup and publishing actions, and other project-wide oXygen settings.

The following additional eXist packages from the HistoryAtState organization on GitLab and GitHub are used on hsg-project systems:

1. hsg-staging (for previewing content before publication)

2. release (for batch generating ebooks, press releases, and other publication assets)

System Usage and Capacity

Since 2011, when HSG began collecting usage statistics via Google Analytics, the site has experienced a ten-fold increase in users and nearly five-fold increase in page views:

To handle this expanded demand, FSI/OH progressively added server capacity, incorporated replication and load balancing, and enabled caching of frequently accessed resources and resource-intensive views. Other adaptations included the addition of testing and development facilities; continuous integration, monitoring, and publication management systems; and integration with redundant external storage and version control repository services (S3 and GitHub/GitLab). Looking ahead, FSI/OH is always seeking opportunities for improved stability and performance of the application.

Usage statistics are not collected for the CMS server, but the current architecture readily sustains the usage of our 50 internal users.

Requirements/Responsibilities

FSI/OH requires enterprise level support in two primary areas, each with its own service level agreement, or SLA: (1) a maintenance SLA, to ensure the smooth and secure functioning of production systems (namely, HSG and CMS), and (2) a development SLA, to create new applications, extend eXist and existing applications with new features, and improve the performance of these systems. A maintenance SLA will be required for all years of the contract. The development SLA is divided into two portions: (2a) specific projects listed below, and (2b) additional projects similar in nature but not yet formulated. Vendors should provide fixed price quotations for completion of the specific projects listed below. The government intends to award the contract for the specified projects to be completed within one year from award, subject to availability of funding. Specific dates will be finalized at the time of obligation. Vendors should also formulate a notional labor hour rate that can be used to provide price quotes for additional development projects. As FSI/OH requires additional projects, FSI/OH and the vendor will use this notional labor hour rate to develop an agreed upon hourly level of effort and a price quote for each project, and separate line item can be added on a firm fixed price basis.

Maintenance Service Level Agreement – general responsibilities

The Maintenance SLA includes all activities related to ensuring the smooth functioning of FSI/OH’s production systems (namely, HSG and CMS).

1. Vendor ensures smooth functioning of production systems.

2. Vendor configures, maintains, and monitors all server instances, as well as applications and services running on the servers.

3. Vendor monitors security advisories and keeps server software updated with security patches.

4. Vendor proactively communicates with FSI/OH to schedule routine maintenance outside of FSI/OH business hours, and to coordinate emergency responses.

5. Vendor provides qualified responses to FSI/OH requests at 4 levels of severity: low, medium, high, and critical.

6. Vendor commits to performing a fixed number of hours of maintenance each month, and availability to perform additional maintenance if requested by FSI/OH.

Development Service Level Agreement

The development SLA includes all other activities related to creating new applications, extending eXist and applications with new features, and improving the performance of systems. The projects listed below must be completed within the base year. FSI/OH may request development services for additional projects that are similar in nature but have not yet formulated at the outset of the contract.

As the vendor completes the development projects listed below and delivers new versions of HSG, CMS, eXist, and the named eXist-based applications and libraries, the vendor must ensure the continued functioning and stability of eXist and all of the core, open source, eXist-based applications and libraries used in HSG, CMS, and FSI/OH’s “hsg-project” development systems discussed above. (These core applications include Dashboard, Documentation, eXide, and Monex, as well as those packages’ dependencies: FunctX, markdown, shared-resources, packageservice, and semver.xq. Also included are packages that FSI/OH systems depend on for proper functioning: TEI Publisher, TEI Publisher Libraries, TEI Simple PM, EXPath Crypto, existdb-saml, Public Repo, Memsort, Messaging and Replication. The vendor must also ensure continued compatibility of eXist’s XML-RPC and WebDAV interfaces with clients such as oXygen XML Editor and Transmit, the functionality of the hsg-project oXygen project file, and the ability of hsg-project users to use the Homebrew system to obtain compatible versions of dependencies such as Java and Node on up-to-date versions of the macOS and Linux operating systems, as used by hsg-project development and server systems.) If the vendor is unable to supply a compatible, fully functioning version of any of these open source applications or libraries (or their successors) for a new stable version of eXist, then the vendor must supply a functional alternative that is acceptable to FSI/OH.

I. HSG development projects

1. Improve HSG search facility:

a. Upgrade the current implementation of the HSG search page to use eXist 5’s facets and fields feature.

b. Migrate existing support existing filters (section, date/time, volumes) and sort options (relevance, oldest to newest, newest to oldest).

c. The definitions of which publications belong to which sections should be easily reconfigurable, to allow changes to site organization and addition of new sections and publications.

d. Match highlighting (KWIC) must work correctly on all eXist’s Lucene-based full text syntax options, including wildcards, boolean, and proximity.

e. Add checkboxes beside keyword search field to enable/disable case-sensitive and/or punctuation-sensitive modes—creating composite or custom full text indexes using eXist’s Lucene-based full text search facility if needed.

f. Add the Administrative Timeline publication (classified under the “Department History” section) to the scope of publications included in full text search.

g. Train FSI/OH staff on how to define and register new facets and publications, assuming a knowledge of XQuery and XML.

h. All query results and associated preparation of accurate, relevant KWIC search highlights should be computed on the server in no more than 500 milliseconds total.

i. KWIC hits for FRUS index pages, which were suppressed due to poor performance and server bottlenecks, should be restored.

j. Each keyword query and filter should be displayed in the search results as a “pill” icon, containing an “x” link which updates the search results to eliminate the criterion from the query.

k. Add new advanced search interfaces for FRUS, Travels, Visits, and Principal Officers & Chiefs of Mission—leveraging eXist’s facets and fields facility.

l. Migrate from the discontinued, legacy TEI Simple library to the current TEI Publisher library.

2. Improve FRUS and historical publications browsing:

a. Each FRUS volume’s landing page should display the volume’s cover image.

b. The Historical Documents landing page should list the most recently published volumes.

c. Create a page displaying all FRUS volumes and availability by format (print, microfiche, electronic-only).

d. From a page image view of a scanned publication, display a link to the corresponding document(s) that appear on the page.

e. In FRUS tables of contents sidebar, indicate where table of contents are truncated and provide a link to expand truncated portions.

f. Fix pop-up footnotes - #202

g. Extend previous/next navigation buttons to other sections, such as current and legacy HAC meeting minutes - #63

h. Show dynamic production chart in FRUS History Appendix A - #155

i. Replace the pages displaying static, screen resolution scans of historical publications with a dynamic, zoomable IIIF-based image implementation.

3. Implement a proper redirect facility for relocated URLs and sections:

a. Background: From time to time, the URLs of resources on HSG are updated, either individually or en masse by section. To ensure readers and search engine bots do not encounter 404 Content Not Found errors, we require a URL redirect facility that allows us to maintain a list of retired URLs and their replacements.

b. Design a declarative format for expressing individual resource URLs and URL patterns and where each should be redirected.

c. Catch requests for matching requests and redirect these requests to the designated pages, using the correct HTTP response for permanently relocated resources.

d. Adapt existing lists of retired URLs to the new system.

4. Implement communication, quality control, and performance improvements:

a. Implement a “report a problem” facility for visitors to report problems with any page on HSG, including typos or broken functionality. Reports should include the URL of the page.

b. Add a “status page” to let readers know when there is a known outage or downtime.

c. Add a “social media card” facility for all HSG pages to include HTML metadata with per-page, per-section, or per-publication images and summary. Allow FSI/OH editors to configure the content of the site-wide default card and define page-, section-, or publication-specific entries.

d. Implement a “link checker” to periodically crawl HSG for links to internal and external resources and produce reports about content not found errors or permanent redirect notices.

e. Create a “sitemap” to aid search engines in crawling our website.

f. Create a “download the raw data” feature to place on all HSG pages to generate links to the appropriate public-facing git repository and source file corresponding to the present view. Create a flexible, declarative template format that FSI/OH editors can populate with the required information.

g. Create a “cite this as” feature to place on all HSG pages to generate MLA or Chicago-style citations, as well as Zotero-compatible citations. Create a flexible, declarative template format that FSI/OH editors can populate with the required information.

h. Create a centralized breadcrumb and sidebar facility - #153 and #161. Create a flexible, declarative template format that FSI/OH editors can populate with the required information for breadcrumbs and sidebar content, which allows definitions to apply to an entire section’s worth of content.

i. Ensure that breadcrumbs and sidebars in Tags section show the section’s full depth - #203

j. Create a new “announcements section.” The HSG homepage features 3 current announcements, but past announcements are not easily discovered. Create a new section, entitled “Blog” or “Announcements,” to house these current and past homepage announcements, press releases, and legacy Tumblr posts.

II. CMS and Publishing development projects

1. Implement a new Git-based publishing workflow:

a. Background: Currently we publish in two steps: (1) we commit and push our completed source files to a remote Git repository, and (2) we upload new or changed files to our production servers using XML-RPC and WebDAV from hsg-project development systems. Editors are responsible for remembering both steps. Also, (3) we perform ebook transformations and convert PDFs to per-page image formats using legacy conversion scripts on our development computers, and (4) we manually upload these and all binary assets to Amazon S3 using the Transmit file transfer client. These processes are labor intensive and error-prone.

b. Implement a publishing system that would allow us to publish content using only step 1, and rely on CI tests and automated routines to confirm XML well formedness and generate derivative media, etc.

c. The editor could preview the resulting resources on a test server. When the editor approves, the updated files would be automatically pushed to the production server and committed to the master branch.

d. Adapt the legacy EPUB generation facility for FRUS publications to use the TEI Processing Model and TEI Publisher—while conforming to FSI/OH ebook style conventions for cover and interior pages, interior section-based tables of contents, and page-based cross reference link resolution. This will bring the benefits of consistency and improve our ability to maintain a centralized code base for TEI transformations to all output media.

e. Adapt the legacy PDF generation facility for generating print-ready FSI/OH research monographs, research papers, style guides, and FRUS documentary editions to use the TEI Processing Model and TEI Publisher—while conforming to FSI/OH print style conventions for cover pages, page dimensions, running headers and footers, footnotes, headings, lists, etc.

f. Implement a static file publishing system using Git LFS or a browser-based application for managing the assets that we currently store on Amazon S3 (i.e., images, PDFs, ebook files), without requiring users to access S3 directly with a file transfer client or manage an AWS API key.

g. Implement automated server-side processes to generate static assets from publications. For example, when updates are performed to TEI files that have associated EPUB- and Mobi-formatted ebook files, the publishing pipeline will generate ebook files for review and, upon approval, publication to the site. Similarly, when a PDF file is uploaded, the publishing pipeline will generate all page images needed for page-based browsing. (Currently all ebooks and static images are generated locally on editors’ computers and uploaded to S3.)

h. In the meantime, continue supporting the existing hsg-project system for setting up local development environments and publishing to production HSG servers.

i. All changes must be tested with a focus on continuity of the key the development and publishing workflows described in the hsg-project repository.

2. Implement browser-based annotation and editing of TEI XML documents:

a. Users must be able to select a span of text in an existing TEI document in a TEI Publisher HTML view, identify the entity type (e.g., person, place, organization, etc.), perform a look up of the selected text against a database of existing individuals (an authority file system), and either select an existing individual from the lookup or submit a new candidate to the authority file system.

b. When a user identifies a span of text as a referring string for the entity, the annotation interface should offer to store that string (or an edited version of it) as a variant.

c. The annotation interface should allow users to apply the same annotation to identical spans of text in the same document.

d. The annotation interface should allow users to trigger a scan of an entire document (or division thereof) against known entities and variants and display candidate markings for possible entity matches.

e. The annotation interface should allow users to perform an NLP-based scan of an entire document (or division thereof) to highlight likely proper nouns that may be entities, even if the entities do not yet exist in the database.

f. The annotation interface should allow users to perform limited editing to the base text, e.g., correction of typos identified within block elements.

g. The annotation interface should also allow advanced users to directly edit the TEI XML source of the selection, to enable edits beyond the scope of a browser-based editor.

3. Implement a taxonomy and authority file system (FRUSpedia):

a. Background: Each type of entity described in the interface above should reside in a taxonomy management and authority file system. The types of named entities we track include Persons, Places, Organizations, Subjects, Sources, Publications, Staff, etc. We have created a notional data model for these entities in Airtable (a web-based relational database management system) but lack a means of managing this data in the eXist-based apps we use to house our documents and data.

b. The authority file system will be the central repository for data named entities in FSI/OH publications and datasets.

c. Editors will be able to propose new entities or edit existing entities. These changes will trigger a review workflow, wherein managers will approve proposed changes.

d. Changes relating to data published on HSG, such as FRUS volume titles and status of the series data, can be published to HSG.

e. The annotation interface described above will draw from the entities in this authority file system.

4. Implement author management facility in HSA:

a. Background: The HSA application contains digitized studies. Each study has an author, and we have a list of authors in XML but no web-based interface for managing these authors.

b. Leverage the entity management database above to house a controlled list of HSA authors and attributes about them.

5. Enhance searching and browsing in HSA:

a. Besides the existing facets, the following facets should be added, based on existing metadata fields in the documents: Years of coverage, Open at NARA, Full text availability, Proofread status,

b. Besides the existing advanced search fields, the following fields should be added: Location, Abstract, External notes, Internal notes.

c. Replace the current codes displayed for Distribution status, Open at NARA, and Tags with labels defined in lookup tables.

d. Add a “# of results per page” drop down to the document table and search results views.

e. Extend HSG punctuation- and case-sensitive options and KWIC search improvements detailed in “Improve HSG search facility” above to HSA.

f. Fix broken links to page images for scanned studies. Use the IIIF facility detailed in “Improve FRUS and historical publications browsing” above to HSA.

6. Enhance the History Mailbox Archive (HMA) application:

a. Add facility for uploading Outlook .msg files to the application. Uploaded binaries should be converted to HTML using the existing Apache Tika-based conversion facility. The binary and converted documents should be committed to version control.

b. Add facility for downloading existing .msg files from the application.

c. Extend the existing Apache Tika-based conversion routine to extract attachments and display image attachments in the application.

d. Bring navigation and styling of application into line with other CMS applications.

e. Bring faceted browsing and field search from HSA: sender, recipient, and date.

7. Develop an Administrative History Portal application:

a. Create a clone of the HSA application and extend it to house and allow ingestion, browsing, and searching of a diverse range of content, including TEI, PDF, and MS Word documents.

b. Extend Location field to include URLs to external resources.

8. Develop an Oral History Portal application:

a. Create a clone of the HSA application to allow ingestion, browsing, and searching of TEI, PDF, and MS Word transcripts of oral histories.

b. Add a field for rights and permissions, with values supplied by FSI/OH.

III. eXist development projects

1. Background: As expert users of the World Wide Web Consortium’s XQuery language, FSI/OH historians depend upon their XQuery implementation to be nearly 100% compliant with the XQuery standard (i.e., the family of XPath and XQuery specifications), as defined by compliance with the XQuery Test Suite. To the extent an implementation falls short of compliance with the specification or contains bugs, FSI/OH historians are less productive and unable to make full use of this language for their research and development projects. Similarly, FSI/OH historians rely on the major eXist-based application and library packages to function properly. These development projects are all urgent priorities for FSI/OH.

2. Implement functions and expressions for full conformance with XQuery 3.1 specification:

a. Implement all remaining XPath and XQuery 3.1 Functions & Operators - #3635

b. Implement XQuery 3.1 tumbling window, sliding window, and count clauses - #3636 and #491

c. Implement XQuery 3.1 item-separator serialization parameter - #1483

d. Implement xs:dateTimeStamp datatype - #1715

e. Implement use-character-maps serialization option - #3237

f. Implement XQuery Update Facility 3.0, both for in-memory and on-database nodes - #3634

3. Fix the following bugs in the eXist core reported by FSI/OH:

a. xmldb:store fails on remote binary resources - #1456

b. XQuery Update ignores namespace binding conflict - #1482

c. util:expand alters node type of attribute and document nodes - #1562

d. AST “unexpected end of subtree” error with map postfix lookup - #1655

e. Bugs in encoding special characters in resource names - #1824

f. Path expression with nested positional predicate fails on database node - #1892

g. Cardinality of return types not enforced - #1917

h. NPE in LuceneMatchListener.scanMatches when expanding node containing both Lucene & NGram matches - #2102

i. XQuery parser chokes on legal attribute names - #2381

j. Incorrect facet count with field query - #3207

k. xs:QName castable as returns incorrect results - #3390

l. div operator error when arg is a variable - #3441

m. RESTXQ infinite loop with circular dependency - #3448

n. “variable is not set” error with mutual imported modules - #3449

o. Erroneous error raised by empty array constructor - #3472

p. missing location info in errors for functions inside element constructors (err:XQTY0105) - #3474

q. Guest users can create collections hidden from dba users - #3505

r. Query only returns results from “new range index” collection, omitting results from other collections - #3620

s. NPE at org.exist.xquery.modules.range.RangeQueryRewriter.rewriteLocationStep - #3624

t. First run of DMG eXist.app on macOS quits before launch - #3637

u. strange regex behavior of fn:contains() - #1379

4. Implement the following enhancements to the eXist core:

a. Adopt lazy loading model for module imports - #1848

b. Expose eXist-specific serialization parameters to standard methods - #2394

c. Include location when throwing all XPathException errors - #3241

d. Integrate XQuery Test Suite into CI for PR checks - #3632

e. Use Saxon for fn:matches regular expression handling - #3633

f. Update to Lucene 5.3+ - #1160

5. Address the following requests by FSI/OH regarding apps included with the eXist distribution:

a. eXide

i. Save as should use current file’s collection - #104

ii. Glitches in final character of XQuery snippets - #105

iii. Login dialog should focus on User input field with File > Manage #109

iv. Add download button to DB Manager view - #111

v. Autocomplete fails to notice that a comment block is already closed - #112

vi. Save keyboard shortcut doesn’t work when focus is in Find pane - #113

vii. Save as does not update the directory sidebar - #114

viii. Add option to disable auto-pair of quotes, parens, etc. - #117

ix. Warn user when saving contents of XQuery window with non-.xq* file extension - #126

x. Switch to xqlint for syntax highlighting of query results - #147

xi. With configuration.xml set to guest=no, login is impossible - #164

xii. Resource operations using “name” instead of “key” are prone to fail #185

xiii. xqlint error causes “run test” to silently fail - #224

b. Dashboard

i. Logging in shouldn’t require duplicate entry of credentials - #156

c. Documentation

i. Detail access details methods for JMX info over HTTP - #106

ii. Update the URL Rewriting article - #200

iii. Document the “exist-serialization” processor instruction - #265

iv. Update the XQuery article to note unsupported serialization parameters - #436

v. Document the “lowercase-expanded-terms” option - #447

6. Implement the following improvements to TEI Publisher:

a. Upgrade EPUB generation facility from EPUB 2 to EPUB 3—specifically for its support of “pop-over footnotes.” The current EPUB 2-based link approach to footnotes is jarring and clunky.

b. Extend TEI Publisher to support the annotation interface described in “Implement browser-based annotation and editing of TEI XML documents” above.