Attachment A SOW.docx

Enterprise Support for eXist

Introduction

The Foreign Service Institute (FSI) is the training bureau of the U.S. Department of State (DOS) and a principal training institution for the U.S. Government foreign affairs community. The Foreign Service Institute is the U.S. Government’s premier foreign affairs training provider, serving the U.S. Department of State and the entire USG foreign affairs community as a strategic enabler of diplomatic excellence. We are dedicated to ensuring the career-long learning opportunities required for success in today’s global arena. The Institute is located at the George P. Shultz National Foreign Affairs Training Center (NFATC) in Arlington, Virginia, with additional training centers domestically, overseas, and in the Washington metropolitan area.

Purpose

The government is considering options to improve its current arrangement for server administration and application development for the existing website, history.state.gov (HSG), and the existing content management system, cms.history.state.gov (CMS), which are both housed in the Amazon EC2 Cloud and administered by the Office of the Historian (FSI/OH). The Office seeks to improve the performance of HSG and CMS, add capabilities to them, and bring several of their components onto internal networks. Any new architecture will need to build on the existing application code written for eXist-db and comply with DOS policies including Cloud solutions approved by DOS’s Cloud Computing Governance Board (CCGB).

This Request for Information (RFI) is being issued solely to determine potential sources for information and planning purposes – it does not constitute a Request for Proposal (RFP) or a promise to issue an RFP in the future. This RFI should not be construed as a commitment by the Government for any purpose other than to gain the information requested for market research. The purpose of this notice is to conduct market research, a continuous process for collecting and analyzing information about capabilities within the market to satisfy agency needs.

Scope

Should FSI decide to proceed, FSI would seek a contractor to provide all computing and support services required to architect and administer software and servers to maintain and improve FSI/OH’s public website, internal applications, and cloud-based server infrastructure.

The contractor will provide ongoing services required to operate, maintain, and safeguard FSI/OH’s existing cloud-based servers and infrastructure and develop further solutions for enhancing FSI/OH’s eXist-based applications (including, as needed, improvements to the open source eXist database software package or other components), and train FSI/OH staff on the use of the applications and infrastructure.

Period and Location of Performance

The base Period of Performance is for one (1) year from the date of contract award with six (6) one-year option periods.

All phases of this effort may be performed offsite.

FSI/OH location:

U.S. Department of State, Potomac Annex (SA-4D) 2300 E Street NW Washington, DC 20372

The contractor shall be responsible for having the appropriate workspace, equipment, and material necessary to provide all deliverables requested by the government. The government will provide needed equipment for remote network. Travel in and/or around the local Washington, D.C., metro area for the purpose of working at the contractor’s place of performance will not be charged to the government.

At the request of the government, occasional attendance at meetings or other events offsite in the local Washington, D.C. metro area may be required. The government will notify contractor personnel of meeting requests in advance of attendance.

Background

The Office of the Historian (FSI/OH), within the Foreign Service Institute (FSI) at the U.S. Department of State (DOS), is responsible, under law, for the preparation and publication of the official documentary history of U.S. foreign policy in the Foreign Relations of the United States (FRUS) series. FSI/OH also prepares policy-supportive historical studies for Department principals and other agencies and conducts an array of other initiatives, ranging from briefing memos to multi-year research projects.

FSI/OH maintains a public website at history.state.gov (hereafter, HSG). In addition to a full text archive of the FRUS series, which contains over 300,000 documents from 550 books published since 1861, HSG contains a history of U.S. diplomatic relations with every state in the world, a database of all of DOS principal officers and chiefs of mission since 1789, and other numerous scholarly resources for the study of U.S. foreign relations and DOS institutional history. In keeping with the Open Government Directive’s principle of transparency, participation, and collaboration as the cornerstone of an open government, all of FSI/OH’s publications can be freely downloaded in the Text Encoding Initiative (TEI) format, an XML-based standard used widely in scholarly electronic text projects. Besides HSG, FSI/OH posts the HSG source code and raw data to GitHub (see github.com/HistoryAtState).

Since its launch in 2009, HSG has been built on a free, open source software project, called eXist (see exist-db.org). eXist excels in the storage and querying of XML data and has enabled the Office to contribute to all key open government initiatives on behalf of the Department, including the Open Government Directive, the Cloud First Initiative, and the Digital Government Strategy. FSI/OH has also used eXist for numerous internal applications, including a content management system (CMS). Staff install eXist and a related suite of software on their workstations to facilitate data analysis, reporting, and development of new applications, publications, and datasets. Because eXist plays such a critical role in the Office’s internal and public-facing digital initiatives, knowledge management activities, and publishing and reporting workflows, FSI/OH requires enterprise level support and development services for eXist, applications built on it, and the server infrastructure housing it.

Current Server Environment

FSI/OH’s public website (HSG), internal application server (CMS), and supporting servers are hosted in the Amazon Cloud (EC2 for compute, ELB for load balancing, and S3 for storage), in Northern Virginia (spread across two availability zones). The EC2 servers are running Amazon Linux (migration to Amazon Linux 2 will be completed soon), with the exception of one CentOS server. Software running on the EC2 servers includes:

· eXist (the core back-end application server and database)

· ActiveMQ (messaging server for database replication)

· Nginx (front-end web server)

· Cantaloupe (image server)

· Ansible (orchestration)

· Jenkins (continuous integration)

· Icinga, Munin, and Cloudwatch (monitoring)

· OpenVPN and Unbound DNS (for secure access to the VPC)

Integrations with external systems beside Amazon Web Services include:

· GitHub.com and GitLab.com (version control)

· Microsoft Azure (DOS-administered single-sign on access to internal web-based applications)

· NPM, Bower, and Maven (package management)

The EC2 servers are allocated as follows:

(See aws.amazon.com/ec2/instance-types for a description of the instance types.)

Each HSG eXist-based backend database server is proxied by a frontend Nginx-based web proxy server, with traffic distributed by Amazon’s Elastic Load Balancer. Data is replicated between the two databases with ApacheMQ, using the Messaging and Replication library for eXist.

All software deployments and upgrades on the FSI/OH servers are orchestrated via Ansible. Automated web tests are performed on a per-commit or nightly basis via Jenkins.

The source code for HSG’s application logic is stored in the HistoryAtState/hsg-shell repository on GitHub, and the underlying publications and datasets (all unclassified, no PII) are stored in approximately 15 other HistoryAtState repositories on GitHub:

· administrative-timeline

· carousel

· conferences

· frus

· frus-history

· gsh

· hac

· milestones

· other-publications

· pocom

· rdcr

· tags

· travels

· twitter

· visits

· wwdai

Required eXist packages not included in the stock eXist distribution are fetched from an eXist-based package repository (public-repo) running on the dos-ci server:

· crypto (An implementation of the EXPath Crypto specification)

· exsaml (for SAML support for authentication with Microsoft Azure SSO)

· memsort (for pre-sorting large datasets; will be discontinued after adopting eXist 5’s facets and fields facility)

· messaging-replication (for replicating eXist between a primary and replica server using ApacheMQ)

· s3 (for interacting with the Amazon S3 API)

· tei-publisher (for transforming TEI XML into HTML and other output formats)

· tei-simple (a legacy version of TEI Publisher)

Of these, TEI Publisher is an especially important part of the application stack. In 2016, HSG migrated from custom XQuery/XSLT for the display of its TEI publications to the open source, eXist-based TEI Publisher library used widely in the TEI community. This library is responsible for the transformation of all of OH’s TEI XML data into HTML, EPUB, and PDF. At the same time, HSG adopted the U.S. Web Design System to achieve its modern, mobile-friendly, and accessible front-end design.

A second public-facing server, labs.history.state.gov, has recently been established to house draft-stage or experimental scholarly resources, stored in the following HistoryAtState git repositories on GitHub.com or alternatively on GitLab.com:

· consular-cards

· diplomatic-list

· frus-indexes

· frus-sources

· frus-unbound

· people

· terms

The CMS server houses two internal applications, Historical Studies Archive and Guides. The application logic and underlying data is stored in private HistoryAtState git repositories on GitLab.com:

· hsa

· hsa-binaries

· hsa-data

· guides

· guides-data

New and updated data is published to the HSG eXist servers using either (1) WebDAV upload of individual files from FSI/OH workstations or (2) Upload of full application/data packages via the eXist Package Manager, accessible only via VPN. The latter method currently requires the replication trigger to be manually disabled and re-enabled when packages are published, but a facility to automate the disabling/re-enabling of the replication trigger is being developed.

The CMS server allows authorized FSI/OH users to edit and upload data. All edits to TEI XML data are synced to HistoryAtState git repositories on GitLab.com, and all binary files are synced to a private S3 bucket. When bulk updates are performed outside the CMS server, they can be deployed to the CMS server via the 2nd method mentioned in the previous paragraph.

Current Desktop Environment

FSI/OH’s standard desktop environment for members of the digital initiatives team is called “hsg-project” (see github.com/HistoryAtState/hsg-project/wiki). These workstations are connected to a dedicated internet network (DIN), which can access the public internet, from which they obtain the required software for the hsg-project environment (software which can also be installed on home computers used for telework).

· macOS 10.15 (Catalina)

· oXygen XML Editor 21.1

· eXist-db 5.2.0

· Transmit 5.6.2

· GitHub Desktop 2.5.3

· Homebrew 2.4.1

· Java OpenJDK 1.8

· Apache Ant 1.10.8

· Apache Maven 3.6.3

· NodeJS 10

· Gulp 2.2.0

· Bower 1.8.8

After installing this software, users clone the hsg-project repository from HistoryAtState, which contains and running included scripts to clone all data and source code repositories and fetch all require packages, users are able to interact with a complete local copy of the HSG website and perform all content editing, data analysis, and publishing tasks.

The following additional eXist packages are used on hsg-project systems:

· hsg-staging (for previewing content before publication)

· release (for batch generating ebooks, press releases, and other publication assets)

System Usage and Capacity

Since 2011, when HSG began collecting usage statistics via Google Analytics, the site has experienced a ten-fold increase in users and page views:

To handle this expanded demand, FSI/OH added server capacity, incorporated load balancing, and enabled caching of frequently accessed resources and resource-intensive views. Other adaptations included the addition of testing and development facilities; continuous integration, monitoring, and publication management systems; and integration with redundant external storage and version control repository services (S3 and GitHub/GitLab).

Usage statistics are not collected for the CMS server, but the current architecture readily sustains the usage of our 50 internal users.

New Target Server Environment

In order to comply with DOS Cloud policies, the FSI/OH Cloud servers must be migrated to a FedRAMP-low compliant environment. FSI/OH has already initiated this process by moving from a 3rd party AWS reseller to the Cloud Project Management Office’s AWS reseller. The vendor will assist FSI/OH in making necessary adjustments to comply with FedRAMP-low requirements.

Requirements/Responsibilities

Servers

1. Vendor configures, maintains, and monitors all server instances and responds to outages.

2. Vendor recommends instance types based on current and expected load, adjusting instance sizes based on actual traffic and usage.

3. Vendor monitors security advisories and keeps server software updated with security patches.

Applications

1. Vendor monitors application performance and responds to outages.

2. Vendor develops new application functions and features as requested by FSI/OH.

3. Vendor ensures Section 508-compliance and mobile friendliness of applications.

Evaluation factors

Vendor must demonstrate direct and relevant expertise and experience in “full stack” web development, with particular expertise in the following technologies or API-compatible equivalent systems:

1. eXist

a. Ability to efficiently and effectively troubleshoot and fix problems in the eXist core. Ability to develop new features and implement new XQuery functions and syntax. Ability to submit changes upstream to the eXist project for community review.

b. Ability to configure eXist for security-hardened production environments.

c. Ability to devise and perform server load and stress tests, and create web integration tests for FSI/OH applications and datasets.

d. Mastery of XQuery 3.1 and eXist XQuery extensions, as well as the XPath, XQuery, and XML family of specifications.

e. Ability to profile the performance of XQuery scripts and eXist libraries.

2. TEI and TEI Publisher

a. Expertise implementing solutions for Text Encoding Initiative (TEI)-encoded XML publications.

b. Expertise using the TEI Processing Model to define and control the transformation of TEI XML into various output formats, including HTML, PDF, and ebook.

c. Experience using the TEI Publisher libraries to develop customized publishing solutions.

3. Java

a. Ability to evaluate JVMs and configure JVMs for memory-intensive production environments.

b. Ability to diagnose and solve problems with JVM configuration and performance.

4. ApacheMQ

a. Ability to configure and maintain ApacheMQ for replicating eXist databases.

5. Apache Ant and Maven

a. Ability to develop build scripts and fix problems.

6. Ansible

a. Ability to develop and maintain orchestration scenarios for cloning, patching, compiling, upgrading, and deploying eXist, eXist-based packages and applications