StreamlinedML.docx

Addendum 1 for BAA FA8750-17-S-7006 entitled, “StreamlinedML”

1. Introduction/Background Machine learning (ML) has the potential to transform how the AF/DoD reasons and acts in complex environments. However, to obtain even a basic learning capability, expertise is required in both ML techniques and the domains in which they are applied. Furthermore, the development and application of modern machine learning solutions for real-world tasks requires substantial investments in the collection of large amounts of domain-relevant data, the implementation of state-of-the-art learning algorithms, and the software pipeline required to configure, train, and evaluate developed learning capabilities.

The current approach within the AF/DoD is to consider each ML application as a separate stovepipe, where ML capabilities are developed independent of one another. This approach has multiple disadvantages. First, it prevents the reuse of data, models, and software pipelines for related learning problems and application domains, resulting in duplicated collection and development efforts. Second, different models for the same problem cannot be directly compared as they lack a common means of evaluation for a given application. Third, such systems are not designed to be open, preventing domain users’, application developers’, and researchers’ full use of the developed ML capabilities and limiting the ability to incorporate new data and techniques as the state-of-the-art progresses.

The goal of this effort is to develop an open, end-to-end ML framework to address these challenges and revolutionize how the AF/DoD designs, develops, trains, evaluates, and deploys ML capabilities. Specifically, this effort is focused on addressing the engineering and research efforts necessary for the realization of a high-performance ML system with emphasis on extensibility, (re)usability, ease of deployment, and scalability. Additionally, this effort seeks to establish an ML ecosystem, built upon the framework, consisting of large-scale data and model repositories, diverse state-of-the-art ML methods, and comprehensive evaluation capabilities to be made available to and built upon by AF/DoD and academia and industry partners. The Government calls the realization of this framework & ecosystem StreamlinedML, as the goal of this system is to streamline the end-to-end process of practically using machine learning for AF/DoD tasks.

2. Problem Description StreamlinedML aims to develop an open, end-to-end ML system to enable the rapid, low-cost deployment of state-of-the-art ML capabilities for AF/DoD data and applications. The goal is to drive the incremental cost of selecting, training, evaluating, and deploying state-of-the-art ML capabilities to AF/DoD applications towards zero.

The StreamlinedML system will require multiple key capabilities to achieve this goal. First, the system should include an initial suite of state-of-the-art learning methods to train models for a number of AF/DoD relevant learning tasks. Second, it should provide a persistent and open repository of data, supervision, and trained models taken from publically available sources, AF/DoD, as well as data uploaded from users of StreamlinedML, and models previously trained using the system. Third, StreamlinedML should be extensible, allowing users to contribute not only new data, supervision, and models, but also implementations of new learning methods. To achieve this, StreamlinedML should utilize open and widely-adopted learning engines, such as but not limited to Tensorflow[footnoteRef:1], Torch[footnoteRef:2], Cafee[footnoteRef:3], and Theano[footnoteRef:4]. Furthermore, the system should expose these to the user through an application program interface (API). The API should make it easy for new learning techniques to be incorporated into the system. Going further, the system should also allow users to easily incorporate new learning engines, as they are developed. [1: https://www.tensorflow.org/] [2: torch.ch/] [3: caffe.berkeleyvision.org/] [4: http://deeplearning.net/software/theano/]

Fourth, StreamlinedML should cater to a range of users from application developers with no ML experience to ML experts. This necessitates an intuitive user interface that considers multiple use cases. In addition, strong emphasis is placed on allowing users to easily explore and compare stored data and models. A user should have the ability to search through the persistent store via meta-data associated with each data set and model. Users should also be able to evaluate the performance of different models, different parameterizations of those models, and how they perform on different data sources by viewing performance statistics provided by StreamlinedML. It is anticipated that this comprehensive approach to evaluation can be used to facilitate model discovery and recommendation for a given task, including suggestions for models that may not have been considered by the user.

Finally, it is anticipated that the StreamlinedML framework will utilize a high performance computing system with sufficient data storage and access to multiple central processing units (CPUs) and graphics processing units (GPUs) either through a single unit or distributed across multiple units with high-performance network connectivity as is typical in a DoD High Performance Computing (HPC) environment.

3. Example System Architecture To provide some clarity into the desired components of the StreamlinedML system, Fig. 1 contains a system diagram that illustrates one potential realization of the system for discussion purposes only. Note the system diagram in Fig. 1 is not intended to provide a comprehensive or final architecture of the system but to instead notionally identify minimal key components and possible component linkages. The figure shows three components that the user can interact with. The persistent store holds all stored data instances and any supervision over those instances, models previously trained using a method from the suite of ML methods and a data set, and statistics chronicling how models in the store performed on a subset of data sets. The persistent store should be able to scale to support large-modern data sets and supervision as well as be extensible enough to easily grow as new data is added or generated.Persistent Store Instances Supervision Learned Models Performance Statistics User Interface Learning Component ML Method Suite Learning Engine Evaluation Component Evaluation Metrics Evaluation Engine Visualization Engine Data Store Active Learning (AL) Component AL Methods Collection Component

Figure 1: Illustrative overview of key StreamlinedML components. System diagram is not intended to provide a comprehensive or final architecture of proposed system but to instead notionally identify key components and possible component linkages.

Users: AF/DoD & Partners

The evaluation component contains metrics that are used to evaluate models, in conjunction with both data sets and models. This engine is meant to evaluate the performance of models on and across data sets with respect to comprehensive evaluation metrics. The output of the evaluation engine is used in the visualization component to present and summarize all metrics, analysis, and statistics on models and data in the system.

The last component is the learning component, which contains the suite of implemented ML methods. Methods are used in the learning engine with data sets to train a model and then store it in the learned model store. Also in the learning component is an Active Learning (AL) framework component, which contains modules for the AL methods and mechanisms to iteratively collect supervision.

It is anticipated that most subcomponents should keep meta-data. For instance, a model in the persistent store should have meta-data that identifies what data and what method was used for training. Models should also contain meta-data indicating what learning problems they are applicable to, what hyperparameters are used for training, and how long it took to train, as examples of potential measures. Keeping such meta-data allows the exploration of both the stored models and data. It also allows users to identify and then select which models and data match their learning problem and application domain. All should be accessible via APIs.

4. Technical Areas The StreamlinedML program is divided into five technical areas (TAs). Proposers may submit offers on any technical area individually or any combination of technical areas thereof. Proposers who include multiple technical areas in their proposal should describe the work for each technical area separately and clearly identify the applicable TAs. Proposers should indicate their integration strategy for integration of components developed in other TAs when applicable.

The initial effort for each TA should be focused on establishing the basic functionality of key components for the StreamlinedML system. Only after these components are realized should additional capabilities be added.

TA1: Machine Learning Framework. This area is responsible for the overall system design, and integration of all components given by other TAs into a single framework from initial inputs to model implementation. The framework should be:

1. Accessible from the collection/ingestion of data and other inputs from the user through to the evaluation and execution of trained models. Emphasis is placed on the execution of models that are easy to use and have the ability to integrate into third party applications. Web-based services and standalone application executables and libraries are two examples of how trained models may be executed by end-users and third party applications.

2. Persistent as to store large amounts of data, supervision, previously trained models, and implementations of learning techniques. Further, meta-data on these elements, as well as statistics on training and evaluation should be kept to support searching and reasoning.

3. Extensible to allow all users the ability to add to the persistent store, and expert users a programming interface to access, extend, and add to the core components of StreamlinedML.

4. Continuous evaluation. As the persistent store grows, meta-data and statistics can be used to recommend specific, pre-trained models for a given problem or data set. This information may be used to continuously compare the performance of different models, hyper-parameters, and data sets. Efficient techniques for parameter selection should be considered (Li, 2016). Further, advanced analytic techniques may be considered to automatically identify new combinations of data sets to train or to retrain new models as the persistent store is expanded and provide analysis.

TA1 is responsible with the creation of the persistent store as well as handling all interactions with all other components. Specifically, TA1 is responsible for integrating all technologies and components developed in TA2-TA5 into the framework. As such, it is highly recommended that TA1 performers fully understand the components of all TAs. Finally, TA1 is responsible for installing and executing incremental snapshots of the framework and integrated capabilities on systems. These snapshots will be used to support evaluation and testing by the Government. It is anticipated that the StreamlinedML framework will function on a DoD-HPC system/network or similar environment.

TA2: Suite of SOTA Machine Learning Methods. An initial suite of state-of-the-art learning techniques is sought to provide a strong and diverse base for users to employ within the StreamlinedML system. These techniques should both support a diverse set of learning tasks of wide interest to AF/DoD and represent the state-of-the-art. Emphasis will be placed on approaches that propose implementation of multiple techniques for one or more learning tasks or domains. For example, the image-to-caption problem has received much recent attention leading to over ten approaches solving variants of the problem (Raffaella Bernardi, 2016). Implementation of multiple competing methods for a given problem/task are encouraged. This TA seeks implementations of multiple current methods for each problem area or task considered. Furthermore, TA2 performers should also implement multiple evaluation metrics for the techniques they provide. For instance, if binary classification methods are provided, so should the ability to evaluate them using established metrics such as accuracy, sensitivity, specificity, precision, recall, receiver operating characteristic (ROC), and/or confusion matrices. These metrics will be used by TA1 and TA4 to evaluate and visualize the performance of TA2 methods.

All techniques implemented in TA2 will be integrated into the TA1 framework by the TA1 performer. However, developed techniques for a given learning task or domain should follow a common interface for configuration, training, evaluation, storing of the resulting model, and all relevant statistics to facilitate integration. Meta-data should also be specified to assist associating each method with problem types and data. Meta-data for implemented methods can include, but is not limited to type of input (imagery, audio, video, general real-valued vectors, etc.), problem type (object detection, general classification, etc.), and solver used. Note that the solver used need not be tied to specific learning methods and vice versa (i.e. different solvers could be used for a single method and different methods can use a single solver - as is common in modern learning engines).

Note that variety is expected and encouraged in proposed methods. For instance, convolutional neural networks (Krizhevsky, et al., 2012) (Simonyan, et al., 2014) (Szegedy, et al., 2015) (He, et al., 2016) have been shown to be successful for complex tasks involving large amounts visual data, but for simpler tasks, linear models such as support vector machines (Cristianini, et al., 2000) (Schölkopf, et al., 2002) may be sufficient. Including both simple, easily-trained methods as well as more complex, resource-intensive methods in StreamlinedML can enable the system or users to choose the appropriate method that matches the complexity and needs of their application.

TA2 proposals should identify and justify the techniques and models proposed for implementation, focusing on state-of-the-art techniques published in top-tier conference and journal venues. TA2 performers should plan to demonstrate correctness of developed techniques through validation on public data and replication of published results. External open libraries or learning engines, such as those listed in Sec. 2, or any general learning packages such as Scikit-learn[footnoteRef:5] should be clearly identified. [5: http://scikit-learn.org/]

TA3: Data-Efficient Learning Methods. The goal of TA3 is the implementation and development of Data-Efficient ML (DEML) techniques. DEML techniques reduce the reliance on large amounts of supervised data, and/or reduce the cost in obtaining supervision. Such techniques are critical for applying ML to AF/DoD tasks where existing supervision and/or access to domain experts for training is extremely limited. DEML techniques may include, but are not limited to, domain adaptation (Daume III, 2006), transfer learning (Pan, 2010), multi-task learning, zero-shot learning (Palatucci, 2009), data set selection, and the principled refinement or pre-training of learned deep neural networks (Erhan, et al., 2010). The full space of existing and new proposed DEML techniques will be considered.

Further, to enable the efficient collection of new training data and to train ML models in an online fashion, development of an Active Learning (AL) component and set of AL methods is sought (Settles, 2010). This includes the capability to iteratively accept supervised data from a user or group of users, update an AL method in an online fashion, provide queries produced by the AL back to the user(s), and iterate until a specified performance or other learning objective is met.

TA3 proposals should identify existing DEML techniques or propose development of new approaches. All techniques implemented in TA3 will be integrated into the TA1 framework. Any AL framework proposal should adequately address the unique learning, scalability, user interaction, and deployment challenges of developing an Active Learning system. Proposed AL work should be aware of existing active and collaborative learning systems such as Next (Jamieson, 2015) and appropriately consider these systems when developing or enhancing AL solutions for StreamlinedML.

TA4: User Interface, ML Workflows, Performance Evaluation & Visualization. The focus of TA4 is the development of a unified user interface that allows users to interact with the TA1 framework and its components (TA2 & TA3). User interaction will be driven by means of workflows. A minimal example set of targeted use-cases are provided in Sec. 5. The developed interface should support the recommended targeted workflows while also being extensible. The interface should provide a means for exploring data, supervision, models, and methods as well their statistics and meta-data. Further, the interface should present users a comprehensive assessment of model performance across datasets and against all relevant techniques and models. Finally, the interface should provide basic but clear presentation of traditional learning performance metrics provided by TA2 and TA3 performers, as well as enable deeper understanding of model failure and success cases through summarization and display of known results compared with predicted results.

The components of TA4 should focus on usability, consistent user interface design, and responsive interfaces consistent with principled user interaction methodologies. In order to maximize usability and access, a web-based approach utilizing current best-practices and technologies for web development is encouraged. As with TAs 2 and 3, the components developed in TA4 will both utilize and integrate with components in TA1.

TA5: Data Curation & Demonstration. This component focuses on the collection and curation of supervised data sets which will be used to both bootstrap data and supervision for use within the StreamlinedML system as well as demonstrate developed learning capabilities. Curated data and supervision should be identified and developed in support of key AF/DoD operational settings, be applicable to modern and challenging learning problems, have a minimal amount of noise in both instances and labels, and contain enough supervised instances to apply large-scale learning techniques on the order of successful equivalent public datasets. Emphasis is placed on the identification and curation of new datasets with wide applicability and learning potential as well as strong operational AF/DoD and partner interest not already provided in the public domain.

5. Use Cases Users of StreamlinedML will comprise of application developers and researchers with various levels of ML expertise. At the minimum, users will be able to identify the learning problem they wish to solve, with the ability to select it from the ones the system can handle. Beyond this, they may approach the system with different sets of inputs corresponding to different use cases. Inputs may include but are not limited to: 1) the desired learning problem 2) a selection of data and/or models from the persistent store 3) a selection of learning method from the suite of ML methods 4) data, models, hyperparameter settings for ML methods, and/or implementations of new ML methods provided by the user. For each use case, StreamlinedML should output different models. In use cases where a single output is requested, e.g. a particular pre-trained model or a newly trained model with a specific hyperparameterization, it is sufficient for the system to output a single model once obtained.

If a single model is not requested, then further user interaction is required. More specifically, StreamlinedML should allow all models compatible for the specified learning task to be explored and compared to one another. The models should be explorable with respect to meta-data pertaining to each model, including what data, solver, and hyperparameters were used during training. The models should also be comparable in terms of statistics, including performance on various data sets, exemplar failure and success instances, and time to train and perform inference. After a user explores and compares models, they can select a model to deploy. In addition, a user should also have the ability to explore, compare, and select stored data sets at the onset while deciding on other inputs. As a result, the user can incorporate data from the persistent store in their learning workflow leveraging existing data and supervision.

Below is a set of targeted use cases that StreamlinedML should handle, organized by what the user specifies or provides the system. Note, that variations of these, as well as entirely different use cases are possible given different inputs into the StreamlinedML system.

1. User specifies task: Existing trained models compatible with the specified task are presented. Performance statistics of compatible models using standard learning metrics as well as introspection into model performance through the development of summarization and visualization techniques for exploring, for example, data used for training and exemplar failure and success instance summarization. User has option to select and use model via web-based API or integrated into third party application.

2. User specifies task & data with supervision: Data and supervision is provided. Building upon (1), the system should recommend existing models given statistics of provided data and supervision. Training or adapting of new models may be recommended and new models trained and hyper-parameters validated using the provided supervision and/or potentially combined with existing supervision. The remaining workflow components should follow (1).

3. User specifies task & data without supervision: No training data is provided. Building upon (1) and (2), the user may be given the option to actively dispatch and collect new training data through the Active Learning component. Alternatively, data-efficient techniques for leveraging existing data and training data may be recommended. The remaining workflow components should follow (1).

4. User specifies task, data, & provides new learning technique: This use case targets the advanced user focused on incorporating a new learning capability into the StreamlinedML system. It is anticipated that the main mode of interaction for this workflow will be through the system API and target advanced users with support for a scripting environment for extensibility.

6. Problem Domains & Data Types The goal of StreamlinedML is to provide a diverse set of learning capabilities spanning a wide range of problem domains and data types. Initial problem domains of interest include but are not limited to 1) image, object, and scene classification 2) object detection and localization 3) image captioning and visual questioning and answering 4) video captioning and activity classification 5) learning tasks involving natural language, including topic modeling, entity resolution, and trend analysis 6) learning over time-varying signals, including time-series classification and prediction 7) learning from audio data, such as speaker identification, 8) learning rare events/anomalies from network, system, or host-level monitoring data. In addition, this effort encourages proposals for other problem areas and data. Proposals should be well motivated by AF/DoD need to include identification of a new data set derived from either government or open sources at scale, new learning problems, with large potential impact to research and operational communities. The proposer should demonstrate unique expertise or capability.

7. Intellectual Property The program will emphasize creating and leveraging open technology and architectures. Proposers should leverage open ML engines whenever possible avoiding dependency on proprietary technologies. As the goal of StreamlinedML is to provide an ML ecosystem for AF/DoD and partners, proprietary technology will be considered only in compelling cases. Make sure to carefully document and explain these reasons in submitted proposals.

8. Data Rights Unlimited data rights of deliverables is strongly encouraged.

References

Cristianini Nello and Shawe-Taylor John An introduction to support vector machine and other kernel-based learning methods [Book]. - [s.l.] : Cambridge university press, 2000.

Daume III Hal, and Daniel Marcu Domain adaptation for statistical classifiers. [Conference] // Journal of Artificial Intelligence Research. - 2006.

David Duvenaud James Robert Lloyd, Roger Grosse, Joshua B. Tenenbaum, Zoubin Ghahramani Structure Discovery in Nonparametric Regression through Compositional Kernel Search [Conference] // ICML. - 2013.

He Kaiming [et al.] Deep Residual Learning for Image Recognition [Conference] // Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. - 2016.

James Robert Lloyd David Duvenaud, Roger Grosse, Joshua B. Tenenbaum, Zoubin Ghahramani Automatic Construction and Natural-Language Description of Nonparametric Regression Models [Conference] // AAAI. - 2014.

Jamieson Kevin, Jain, Lalit, Fernandez, Chris, Glattard, Nick, Nowak, Robert NEXT: A System for Real-World Development, Evaluation, and Application of Active Learning [Conference] // NIPS. - 2015.

Krizhevsky Alex, Sutskever Ilya and Hinton Geoffrey Imagenet classification with deep convolutional neural networks [Conference] // Advances in neural information processing systems. - 2012.

Li Lisha Li, Jamieson, Kevin, DeSalvo, Giulia, Rostamizadeh, Afshin, Talwalkar, Ameet Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization [Conference]. - [s.l.] : Arxiv, 2016.

Palatucci Mark, et al Zero-shot learning with semantic output codes [Conference]. - [s.l.] : Advances in neural information processing systems, 2009.

Pan Sinno Jialin, and Qiang Yang A survey on transfer learning [Conference]. - [s.l.] : Knowledge and Data Engineering, IEEE Transactions on, 2010.

Raffaella Bernardi Ruket Cakici, Desmond Elliott, Aykut Erdem, Erkut Erdem, Nazli Ikizler-Cinbis, Frank Keller, Adrian Muscat, Barbara Plank Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measure [Conference] // Arxiv. - 2016.

Roger Grosse Ruslan Salakhutdinov, William T. Freeman, Joshua B. Tenenbaum Exploiting compositionality to explore a large space of model structures [Conference] // Conference on Uncertainty in Artificial Intelligence. - 2012.

Schölkopf Bernhard and Smola Alexander J Learning with kernels: support vector machines, regularization, optimization, and beyond [Book]. - [s.l.] : MIT press, 2002.

Settles Burr Active learning literature survey [Conference]. - University of Wisconsin, MA : [s.n.], 2010.

Simonyan Karen and Zisserman Andrew Very deep convolutional networks for large-scale image recognition [Conference] // arXiv preprint arXiv:1409.1556. - 2014.

Szegedy Christian [et al.] Going deeper with convolutions [Conference] // Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. - 2015.