Statement of Work.pdf

Statement of Work FAA William J. Hughes Technical Center

Atlantic City, NJ 08405

ANG-E25 Human Factors Research and Development Branch ANG-C55 Modelling and Simulation Branch

GeoMesa Framework over Amazon Web Services Elastic Map Reduce Training

Table of Contents Introduction

Scope

Contractor Requirements

Curriculum and Content

Contractor-provided Resources

Contractor Qualifications

Government Requirements

Period of Performance

Introduction The purpose of this statement of work is to provide training on the GeoMesa framework for spatio-temporal processing to the staff and personnel of the Human Factors Research and Development Branch, ANG-E25 [1] and the Modelling and Simulation Group, ANG-C55 [2]. The Human Factors Research and Development Branch employs scientific methods and advanced technology to perform research to assure that human operators of systems perform as safely and effectively as possible. The modelling and simulation branch supports Next Generation Air Transportation Systems, new technologies and procedures, and infrastructure. ANG-C55 utilizes modelling and simulation, engineering practices, statistical analysis, and advanced computing methods. A recent combined project has been identified by the ATO Human Performance Team, AJI-342 to study the effects of various levels of workload and the effects of fatigue that Air Traffic Controllers (ATC) may experience as a result. This study is to be conducted using Big Data architectures employed on Amazon Web Services (AWS) GovCloud [3] using data provided within the FAA Enterprise Information Management (EIM) platform.

Workload fatigue is often highly correlated with measures of the air traffic environment, to include air traffic count metrics, hand off delays, separation between aircraft, and the number of convergent and divergent flights. These metrics along with others are required to be calculated over long periods of time and vast spatial extents. It is within the mission of these branches to identify technologies that are available, practical, and compatible with these systems to accomplish the goals of the project. The GeoMesa open source framework [4] has been selected to provide spatio-temporal data processing capabilities at a Big Data scale overtop the AWS GovCloud and EIM platform to accomplish the goals of the ATC Workload project. Training customized and specific to the GeoMesa framework will be critical to the success of this project. The training must cover the best installation and configuration methods, data indexing and ingestion techniques, spatial and temporal queries, data reduction persistence, and endpoints for ad-hoc analysis using open source frameworks and browser-based technologies and tools such as GeoServer, Jupyter Notebooks, Vegas, and Leaflet.

Scope Conduct a six (6) day training session which is broken up into 3 individual 2-day sets over a period of 3 weeks on the GeoMesa framework as configured over AWS Elastic Compute 2 (EC2) and Elastic Map Reduce (EMR) virtual machine instances for up to 12 participants, held virtually. The training will provide the Human Factors Research and Development Branch ANG- E25 and the Modelling and Simulation Branch ANG-C55 with the skill and knowledge to create and administer AWS EMR instances, conduct batch processing of workload metrics, troubleshoot and perform ad-hoc analysis, and produce high quality reports from derived data.

The participants have previous software development experience with big data systems specific to AWS EMR and should be considered to have intermediate level expertise. A portion of developers will have had a large amount of experience in software development in the support of modelling and statistical studies, but may not have had experience with big data and Apache Spark.

Contractor Requirements The Contractor will provide virtual training to FAA-affiliated participants designated by the branches at a specific date to be determined based upon mutual agreement between the FAA and the Contractor, estimated to be between July 5th 2021 and August 30th, 2021.

The virtual training course will be conducted in groups of (2) consecutive days over three consecutive weeks (3), Tuesday and Wednesday of each week during normal business hours, 8:00 am to 4:30 pm with a half-hour break for lunch. The break in weeks is to provide training participants the chance to employ some of the skills acquired and develop specific questions with respect to the content of the course to be asked within the first one to two hours of training at the beginning of the subsequent two-day training block.

Curriculum and Content

The training course must include the topics listed below. These topics are broken out by day, but if the contractor wants to more specifically cover or spread the content into different days that is acceptable provided all of the material is addressed during the course. This curriculum and content may be modified based on mutual agreement of both the contractor and the FAA.

Day 1 and 2 It is envisioned that the first two days would be spent on giving a high-level overview of GeoMesa, establishing an environment on Amazon EMR, and administrative / Devops functions.

• An introduction and overview of the GeoMesa Framework and Architecture o Design considerations o Datastores

Types (FileSystem, HBase, Accumulo) Tradeoffs and performance implications o Indexing methods and optimal approaches o Data Ingestion Techniques with focus on ingestion from S3 buckets

• A demonstration of establishing an AWS EMR cluster utilizing GeoMesa o AWS Configuration, scripts, and steps specific to GeoMesa configuration o Ingestion of ADS-B or telemetry data into AWS EMR cluster using the filesystem datastore over HDFS or S3.

o Running a batch processing job using built JAR.

o Teardown, update, and persistence of data configured using Geomesa.

• AWS EMR Configuration utilizing Jupyter Notebook o Establishing Jupyter Notebook environment with Toree kernel utilizing scripts as part of environment setup employing a supervisor service.

o Basic Leaflet and Vegas visualizations to ensure the service is operating.

• An introduction to local developer workflows using the GeoMesa framework. (*) o Establishing a sample project configured with the Simple Build Tool (sbt).

o Building a local project, bundling compiled code and resources.

o Unit testing and integration testing strategies local to the developer workstation.

NOTE:

(*) The local developer workflows will focus on dependency configuration and build process rather than on the specifics of development such as analytic methods or user-defined functions.

It is envisioned that any issues or challenges in establishing a local development workflow could be addressed by class attendants and support from the contractor via e-mail or remotely between the Day 1 and 2 session and the Day 3 and 4 session when the specifics of GeoMesa over Apache Spark would be discussed in detail.

Day 3 and 4

• A re-familiarization and overview of Apache Spark o User Defined Functions o Resilient Distributed Datasets (RDDs) o Spark Dataframes o Broadcast Variables o Partitioning o Query Plans o Monitoring execution of submitted DAG within Spark UI o Collection and Persistence of Data to Datastores

• Basic to Intermediate GeoMesa Spark o Architecture with focus on the FileSystem Datastores (S3, HDFS) o GeoMesa Spark Core, Spark JTS o Spark SQL o GeoMesa PySpark

• Running GeoMesa Applications on an EMR cluster o Submitting assembled Scala JAR with GeoMesa to EMR Cluster from local file system or S3.

o Submitting PySpark application over python

• Jupyter Notebook over JupyterHub o Running Scala code over Toree Kernel o Running PySpark applications over Python3 kernel o Generation of summary spatio-temporal statistics over jupyter notebook session o Advanced Vegas and Leaflet visualizations.

Day 5 and 6

• Advanced Queries and Applications of GeoMesa Spark o Optimizing spatial and temporal joins o Joining non-spatial data with spatial data o In-memory index impact on performance

• Command Line Tools for ad-hoc Analysis and Configuration

• Data Format Conversion Tools

• Methods and Best Practices for Logging Configuration

• GeoServer Configuration and Producing Endpoints o Examples using FileSystem Datastore o WMS and WFS services

• GeoMesa Processes and examples o DensityProcess o StatsProcess o QueryProcess o TubeSelectProcess

The Contractor must provide Hands-on exercises and examples at a minimum covering:

o Retrieving a set of temporal tracks using the arbitrary polygonal and altitude bounds over a given time period.

o Applying parameterized stereographic projections to aircraft surveillance track data.

o Calculating the closest point of approach for multiple aircraft tracks within a spatial and temporal extent.

o Calculating a number of tracks present within a spatial and temporal extent.

Contractor-provided Resources

The Contractor must provide training solutions and resources including:

• Course itinerary planning.

• Based on mutual agreement of the contractor, the FAA, and course participants recordings of the virtual training may be made available by the contractor if the virtual teleconferencing platform permits.

• An available git repository of local sample project workflow accessible to course attendants.

• Required software and resources for training participants’ local developer workstations.

• Location of pre-requisite installation instructions and support delivered via e-mail prior to training sessions.

• AWS EMR cluster which:

o Is composed of at least one master node and three worker nodes with enough performant power to provide examples of execution that can scale in a production environment.

o Provides end-points accessible to course attendants which can push locally built resources during the days of the training sessions.

o Is available to participants during the training sessions.

o Has all configuration scripts and instantiation details documented and provided to course attendants to replicate the environment inside of their own AWS account.

• Software requirements for the course where dependencies and tools are targeted toward open source software without for-cost license restrictions (IntelliJ- or Eclipse IDE, resources available via the web).

• Digital copies of slides and training resources.

The contractor will recommend and be familiar with the content of a Massive Open Online Course (MOOC) platform (such as edX, Coursera, Data Camp, or other platform service) course covering an introduction to Apache Spark material, and assume the least skill set of developers are those who have completed this course and are very familiar with the material covered.

Contractor Qualifications The contractor should have in-depth knowledge and significant history to include one or more completed professional projects utilizing the GeoMesa framework for the United States Government, and previous experience in providing instructional training in GeoMesa software development in a virtual setting. The contractor should indicate their experience on their submission. The resume, credentials, or description of prior experience of the instructor relevant to the proposed training should be included with the proposal submission. These items will be evaluated based on the experience level with the above training subject matter.

Government Requirements The Human Factors Research and Development Branch ANG-E25 and the Modelling and Simulation ANG-C55 will provide anonymized data samples and historical scripts of exploratory analysis conducted utilizing geospatial frameworks in the past as one available input to exercises and data samples. If available and permitted by organizations with whom the contractor has pre-existing relationships or utilizing publicly available sources, data specific to telemetry or spatio-temporal track data should be used to construct examples for use during training sessions.

Period of Performance The Period of Performance is estimated to be between July 5th 2021 and August 30th, 2021 in three consecutive weeks, and will be determined based upon mutual agreement between the FAA and the Contractor.

Contractor-provided Resources