COG_Pack.pdf

Public Affairs Clearance # 88ABW-2019-4175 (Approved 27 August 2019)

Cognitive Operations Gear (COG) Pack™ Software API White Paper – An API Designed to Enable the Integration of Multi-modal Data-types for Real-time Analysis.

Abstract

Physiological, behavioral, and environmental sensing can provide insight into the moment-to-moment cognitive state of an Airman. In the consumer market, devices capable of capturing these data-types are becoming ubiquitous. As human physiological monitoring capabilities increase in conjunction with the number of available devices, a new set of challenges arise within the software development lifecycle. Those wishing to leverage these technologies to determine and understand human performance are limited in the available solutions. Many steps are required to complete the analysis process. Digitizing the analog signal, aggregating the signals from multiple devices and multiple Airmen, and sending the signals to intelligent algorithms distributed across a network of computers are just a few. However, before implementing a solution, many details should be considered. These take the forms of device identification, inter-process latency, and scalability. Each may cause significant issues if the software is not architected correctly.

Problem Statement

There is no standard off-the-shelf method to synchronize multiple data types from multiple devices connected to multiple people. Each heart-rate cardiovascular monitor, eye-tracker, or EEG device uniquely aggregates its own data. Often, each manufacturer implements its solution differently, and only supports data collected from a single individual for its device. When tasked to collect data from multiple people, the problem space becomes even more complicated. Additionally, if the desired outcome is a real-time assessment based on machine learning or other advanced statistical processes, organizing and defining the data are of the utmost importance. The architecture proposed by COG Pack™ seeks to provide a standard method and approach to solve this problem.

Background

Historically, software applications transmit and store every bit of information any device can capture (Boulay 2018). These solutions usually are geared toward post-processing and analysis of the data requiring months or even years to complete. Often, data is logged in a proprietary file format, then viewed/analyzed with additional proprietary software packages. Other times, the data capture solution fails to integrate all available data-types. These approaches present complex challenges when each manufacturer provides a custom solution. These approaches can be cumbersome to work with at scale and require multiple user interface interactions to manage the data.

COG Pack™ focuses more on describing the signals, rather than all the data any particular device may provide. With this approach, a translation layer has been designed to aggregate only the signals of interest. This alternative methodology permits swapping devices that generate similar signals, say, as old ones become obsolete and new ones become available. Inherent in the design is also the ability to expand and allow the addition of new signals. Additionally, a standard NoSQL database format is implemented for logging the data, thus providing a straightforward backend storage solution and support for universal data access protocols.

Solution COG Pack™ addresses this disparate data-type integration problem by providing a hardware abstraction layer via an API that outlines the data-types of interest. This abstraction layer can then provide various analysis operations with a common data-type to perform additional analytics, regardless of the manufacturer's original data definition. This architecture implements a structure for the sensors, signals, features, and states that co-exist in this type of heterogeneous system.

Signals A Signal is the nomenclature used to describe every event or action. COG Pack™ transmits all the messages on its infrastructure as a Signal sent by a Sensor (the source of the Signal). This Signal contains key properties that correspond to the unique attributes of the Signal and describe the meta-data. These properties are the set of descriptors necessary to support the analysis techniques in COG Pack™ (Figure 1). Signals provide a more generic representation of the data when compared to other data logging systems. For example, physiological Signals include electrocardiogram (ECG) Waveforms, EEG waveform, heart rate, eye-gaze direction, etc. A Signal can also be used to capture behavioral events like mouse movement, keystrokes, programmatic stage completion, observed events, or any other task event.

Figure 1 - Signal Properties

Channels Each Signal in COG Pack™ contains a list of Channels. These Channels are the "meat" of the Signal as they contain the actual values of the data corresponding to different aspects of the Signal (Figure 2). For example, a Sensor that captures values for acceleration would provide channels x, y, and z corresponding to those values of the accelerated state.

Figure 2 - Channel Properties

Sensors COG Pack™ defines a Sensor as the source of the Signal. Every software process residing in COG Pack™ acts as a Sensor (Figure 3). One type of Sensor manages the interaction with hardware, providing the connection between a physical device and the infrastructure. Another type of Sensor is a machine-learning service that receives messages and generates features as new Signals.

Figure 3 - Sensor Properties and Methods

Microservices Architecture

The impetus for the COG Pack™ data transmission infrastructure draws from the computer science industry's push to microservices (MuleSoft. 2018). To achieve the scale described in the problem statement, a distributed system of software services is the best approach.

Starting at layers 6 and 7 of the OSI model (Figure 4.), COG Pack™'s API aims to define the data structures "closest to the end-user" (Shaw 2018). Typically, these layers handle processing and manipulating the data which are of most interest to the machine learning and statistical processes.

Signals, Sensors, and Channels are each layer 6 and 7 defined types.

Figure 4 - Layers of the OSI Network Model

Additionally, distributing the workload amongst individual computers helps minimize the impact of aggregation and analysis on any single system. Thus, a layer 4 or 5 solution is necessary to move the data through the infrastructure between separate software processes. Currently, NetMQ/0MQ, RESTful APIs, and SignalR are implemented to handle the complex process of sending/receiving the data types defined in the higher layers. Formatting the data for transmission is leveraged by using JSON and/or MessagePack formats.

Conclusion COG Pack™ provides a method to integrate hardware, software, and algorithms useful in analyzing human physiological and behavioral data in real-time. Its approach to structuring the sensors, signals, features, and states provides a new and novel definition of each respective data-type.

References

Boulay, Chadwick. (2018) [LabStreamingLayer Wiki]. Retrieved from https://github.com/sccn/labstreaminglayer/wiki

MuleSoft. (2018) The Top Six Microservices Patterns [PDF File]. Retrieved from https://www.mulesoft.com/lp/whitepaper/api/top-microservices-patterns (Sign-up required)

Shaw, Keith. (2018) The OSI model explained: How to understand (and remember) the 7 layer network model. https://www.networkworld.com/article/3239677/the-osi-model-explained-how-to-understand-and-remember-the-7-layer-network-model.html

Glossary COG Pack™ - Cognitive Operations Gear Pack™ Feature – A Signal derived from other Signals Microservices – a collection of loosely coupled services.

Sensor – The entity that generates signals Signal – The data-type that defines a collection of channels State – The end result of the analysis of Signals and Features.

Channel – The data-type that defines the value of a signal https://github.com/sccn/labstreaminglayer/wiki https://www.mulesoft.com/lp/whitepaper/api/top-microservices-patterns https://www.networkworld.com/article/3239677/the-osi-model-explained-how-to-understand-and-remember-the-7-layer-network-model.html https://www.networkworld.com/article/3239677/the-osi-model-explained-how-to-understand-and-remember-the-7-layer-network-model.html