Amendment0002 - D2DTestbed_IndustryDays_.pdf

D2D TestbedD2D TestbedD2D Testbed D2D Testbed

Sanjeev Mohindra and Julie MullenSanjeev Mohindra and Julie Mullen MIT Lincoln Laboratory

DISTRIBUTION STATEMENT A. Approved for public release; distributed is unlimited

This work is sponsored by the Department of the Air Force under Air Force contract FA8721-05-C-0002. Opinions, interpretations, Testbed - 1

SM 3/11/2011

MIT Lincoln Laboratory conclusions and recommendations are those of the author and are not necessarily endorsed by the United States Government.

Outline

• Introduction

• Testbed Overview

• Testbed Architecture

• Initial Testbed Rollout

• Testbed Access

• Summary

MIT Lincoln Laboratory Testbed - 2

SM 3/11/2011

Data to Decision Testbed

D2D testbed is a government-owned prototype of the Data to Decision platform

D2D testbed is a government-owned prototype of the Data to Decision platform

MIT Lincoln Laboratory Testbed - 3 the Data to Decision platform.the Data to Decision platform.

Data to Decision Platform

Data, Apps, Software Infrastructure Distribution

CommunityHardware D2D Platform

Infrastructure Utility Computing

High Performance Computing

D2D Testbed hosts data and applications and

MIT Lincoln Laboratory Testbed - 4

SM 3/11/2011

D2D Testbed hosts data and applications and provides storage and compute resources

Scalable Infrastructure G id

Multiple Parallel File

LLGrid

Single Disk

Multiple Disks System

Storage

Cluster

Grid Cloud g

Single Machine

Cluster

Computation

Cluster(s) TX-X TX-DoD

Classification External Restricted Compute Nodes 120 35 TFLOPs 3 1 0 9TFLOPs 3.1 0.9 Central Storage (TB) 14 8 Distributed Storage

(TB) 9 12

MIT Lincoln Laboratory Testbed - 5

Infrastructure supports storage and computation on large data

• Testbed Overview

• Testbed Architecture

• Initial Testbed Rollout

• Testbed Access

• Summary

MIT Lincoln Laboratory Testbed - 6

D2D Testbed Layer Diagram

Analytic Layer

User Interaction Layer Libraries of UI modules

Data Management Layer

Analytic Layer Libraries of analytic modules

Data Management Layer

• Service-Oriented Architecture

• High Performance

• Scalable

RKA D2D Presentation

Testbed users should be able to access data; buildTestbed users should be able to access data; build and consume analytic services; and interact with data and services.

MIT Lincoln Laboratory Testbed - 7

Testbed Management

Development Testbed

Mature and Tested Apps can be transitioned into the D2D “Production” Testbed

Mature and Tested Apps can be transitioned into the D2D “Production” Testbed

Test on “restricted data”

Test on “restricted data”

(Open) Production TestbedProduction Testbed

Development Testbed

•Selected non-sensitive data

•Open Apps •Data and App

(Restricted)

D2DData and App Interfaces

•Collaboration tools •Selected data •Apps •Data and App interfaces

“Production” Testbed

•Collaboration tools •All data •Selected apps •Data and App interfaces C ll b ti t lConsortium drives continuous

MIT Lincoln Laboratory Testbed - 8

SM 3/11/2011

•Collaboration toolsConsortium drives continuous improvement and transition

• Testbed Overview

• Testbed Architecture

• Initial Testbed Rollout

• Testbed Access

• Summary

MIT Lincoln Laboratory Testbed - 9

Types of Testbed Users

• Develops analytics in Matlab

• Knows that SOA exists

Analytics Developer Analytics

Developer Initial Focus Initial Focus

• Programs in Java

• Loves SOA

Software Developer Software

Developer

S tS t • Hosts testbed

• Makes apps work together

System Integrator System

Integrator

• Analyzes data to make decisionsA l tA l t • Analyzes data to make decisions using apps AnalystAnalyst

Architecture should keep the needs of various users in mind

MIT Lincoln Laboratory Testbed - 10

Data as a service

• Users can mount data on their client machines

• Apps can access data as if it were localApps can access data as if it were local

U t d t th h l l t i• Users can request data through local or remote service apps

• Apps can access through other data service apps

• Data, and data services can be at different locations

MIT Lincoln Laboratory Testbed - 11

Provides users with access to data hosted on testbed

Analytics as a service File Services

Storage

Data Server

Data ServicesAnalytic Services

Web Service

UI

DataData

Server

Database Server

Service

Web Service

• Analytic services interact with data directly or through data iservices

• Client user interaction apps interact with analytic apps

• Users can run analytic apps locally or remotelyy pp y y

• Running analytic apps on the testbed provides data locality and scalability

MIT Lincoln Laboratory Testbed - 12

Users can build and consume analytic services

Matlab Analytic Server

Storage

Data

Data ServicesAnalytic Services

DataData

Server

Database Server

Web Service

Matlab Analytic Server

• Analytic Developer builds apps in Matlab

Server

• To convert to analytic services:

1. Write a Matlab calling function and simply place the Matlab code at a known location

2. The Matlab calling function is automatically exposed as a web service, that can be called using RESTful web service calls

MIT Lincoln Laboratory Testbed - 13

Analytic developer can focus on algorithms

• Testbed Overview

• Testbed Architecture

• Initial Testbed Rollout

• Testbed Access

• Summary

MIT Lincoln Laboratory Testbed - 14

Initial Testbed Rollout

Storage Data Services§

•Image serverSecure Connection

CLIENT MACHINE TESTBEDFor Analytic Developer

DataData Data§

• Bluegrass data

• Other data

Mount Data remotely

Collaboration Version Control

A l ti A §

Data Services§

• Image server

Cluster(s) TX-X TX-DoD

Classification External Restricted Compute Nodes 120 35 TFLOPs 3 1 0 9

Analytic Apps§

• Region tracker

• Entity tracker

API TFLOPs 3.1 0.9 Central Storage (TB) 14 8 Distributed Storage

(TB) 9 12

API

• Region tracker

• Entity tracker

• Image server

MIT Lincoln Laboratory Testbed - 15

§Some data and apps only made available on the restricted testbed

Recommended Practices

DataData

Step 1

• Mount data

• Develop applications on local machinep

Step 2

• Build and test services on local machine

Step 2

Step 3

• Parallelize applications

• Test and deploy on testbedStep 3 Test and deploy on testbed

MIT Lincoln Laboratory Testbed - 16

Don’t let these get in the way of progress

• Testbed Overview

• Testbed Architecture

• Initial Testbed Rollout

• Testbed Access

• Summaryy

MIT Lincoln Laboratory Testbed - 17

Initial Testbed Rollout

Secure Connection

CLIENTS For Analytic Developer

Mount Data remotely

• Requesting an Account

• Accessing the Testbed

Cluster(s) University Government

Classification External Restricted Compute Nodes 120 35 TFLOPs 3.1 0.9 Central Storage (TB) 14 8

• Use Policies

Central Storage (TB) 14 8 Distributed Storage (TB) 9 12

MIT Lincoln Laboratory Testbed - 18

Requesting Access to the D2D Collaborative Environment

P d t• Proposed user must

– Complete Account Request Form which asks for:

User’s Name User’s Status (e.g. Consultant, Subcontractor, Student) User’s Citizenship User’s company or organization P i d f ti th t t d t b tiPeriod of time that account needs to be active

– Not share the account with colleagues or students

– Restrict their cluster use to program related work

T tb d t ill id• Testbed team will provide

– Datasets

– Image and Analytical Web Service Interfaces

– User Guides for

Accessing and using the Testbed cluster Using the Testbed infrastructure to develop services

MIT Lincoln Laboratory Testbed - 19

Accessing the Testbed

Connect via ssh Requires:Requires:

•User Name •Pin •SecureID

XXXXXX

MIT Lincoln Laboratory Testbed - 20

Mount the Testbed File System in Your Local Workspace

XXXXXX

XXXXXX

XXXXXX

XXXXXXXXXX

Mount Testbed File system on local machine

Drag & drop files

Run services

MIT Lincoln Laboratory Testbed - 21

• Testbed Overview

• Testbed Architecture

• Initial Testbed Rollout

• Testbed Access

• Summary

MIT Lincoln Laboratory Testbed - 22

Summary

• Testbed architecture provides a way for users to access data; build, consume and host analytic ; , y services; and interact with data and services.

• Analytics developer can focus on developing algorithms, and leave the SOA aspects to the testbed middleware

• The final testbed architecture can be hosted at any• The final testbed architecture can be hosted at any government team member site.

• Testbed will be online before the performers areTestbed will be online before the performers are selected

MIT Lincoln Laboratory Testbed - 23