DTRIAC Industry Dec 2019 (Presentation).pdf

• Through our revised DTRA mission statement, we retain our focus on DTRA’s three core missions:

1. Supporting the U.S. nuclear deterrent

2. Countering WMD threats

3. Countering improvised threats

• We place particular emphasis on DTRA’s ability to enable our stakeholders, particularly at the Combatant Commands.

• In order to meet the direction provided in the NDS, we are also shifting our focus to ensure that we facilitate competition below the level of armed conflict through countering WMD and Improvised Threat networks.

This slide provides an overview of how all the capabilities discussed earlier link together into a cohesive counter threat networks approach.

At DTRA we enable DoD and the USG to counter and deter WMD and Improvised Threat Networks through the application of the following methodology:

• We work with partners to identify opportunities to bring integrated cross‐domain effects against the threat. Ideally, these are brought as far left as possible – well outside traditional DoD mission space. (Opportunity Analysis)

• We provide tools and analytic support to understand the evolution of the threat, and the connected system (Network Analysis)

• The enemy understands the critical nodes within his own network and takes measures to protect them such as hardening or burying them deeply in the ground. We analyze these targets and develop tailored weaponeering options to hold these nodes at risk.

• Research and development plays a critical role both in understanding potential threat vectors as well as leveraging cutting‐ edge technology to develop options to defeat critical nodes in the threat pathway to proliferation (Pathway Defeat)

• Our partner capacity building programs act as force multipliers for US efforts to counter WMD and IT networks. To effectively counter these dispersed, sophisticated networks, we need a broad network of our own — and international partners are a key part of that (BPC).

• We enable U.S. nuclear deterrence

• Ensure that nuclear infrastructure can operate through an ever increasing set of threats ‐ cyber, physical, EMP;

• Grow the next generation nuclear workforce.

• Targeting and analysis (both conventional and nuclear)

The DTRIAC Task Areas (and CLINs) will be organized into 3 broad Task Areas:

‐ Core Operations ‐ Nuclear Information Analysis Center (IAC) Services

‐ As the DoD Nuclear Deterrent IAC, these option tasks leverage the DTRIAC Collection, Team, and IT Systems to provide dedicated mission or Agency support

‐ Advanced Search and Discovery (ASD) R&D ‐ Development of new capabilities, e.g., OCR or AI/ML Methods, transitioned on a regular basis to Core Operations for routine execution ‐ ASD Analyses will drive improvements to STARS and DTRIAC end‐user support ‐ ASD methods and data will be applied to support other missions and programs ‐ ASD and Core Operations coordination critical – DevOps

‐ ASD development and analyses, including OCR, AI, and ML methods development, will have the following inputs:

‐ Raw data in the Data Lake ‐ existing training data from STARS, as exported to the Data Lake ‐ SME‐based training data, resourced and managed as a Dedicated Support

Option Task

Verification, Validation and Accreditation (VV&A) is fairly recent requirement for all Models and Simulations developed or used by DTRA. DTRA Reachback is working towards accreditation of its tools, with several efforts completed. Archival, search/discovery and access to the documents and data associated with these accreditation efforts provides considerable value, for example, for tool re‐use or future VV&A efforts of the same or different tools sharing underlying engineering codes.

Since CBRNE tools accredited by DTRA may contain as many as 40 or more component engineering codes, organization and presentation of the material for reviewers is challenging. VV&A documentation “packages” may include hundreds of documents in variety of formats using a multi‐level folder structure that is currently presented to the reviewer using one or more overview documents linking to other documents via hyperlinks. Packages may include, as examples, Word and PDF documents, PowerPoint presentations, spreadsheets, Outlook .msg files, .zip files, text files, various image formats, and native tool file formats.

Dozens of test series sponsored by DTRA and its partners, each including many individual test events, provide the data used to validate many of these models. This data exists in a variety of formats; examples are listed in the PWS. DTRA personnel continue to seek sources for even more data. Much of this data is currently pigeonholed with SMEs, DTRA sponsor program managers, etc. If a means to easily archive V&V data were established, it would be discoverable and accessible to programs needing to validate similar codes.

The CBRNE Tools VV&A Resources Task seeks solutions for hosting of and access to VV&A documents and data. Documents and data are frequently shared with DTRA partners and collaborators, so accessibility outside of DTRA networks is a requirement, through CAC‐based access or other means.

Slide 24

MJC41 Added some graphics McAninch, Jeffrey CIV, 11/10/2019

Here are a few examples from the DTRIAC collection. These unclassified documents as messy as they look, are actually in better condition and present more readability than other types of documents. Utilizing OCR for the DTRIAC collection is wrought with issues, and the problem is not easy to solve. The DTRIAC collection while as a whole, contains information that is still pertinent today for scientists, it does not change the fact that the information is of archival age.

The collection dates to the Manhattan Project and even documents from the 1970s and 1980s, were all written on typewriters. Typeface was not considered when OCR standards were developed. For this reason, it is common to see results like on the Shelter Manual on the left. There is a lot of “noise” on the page, which is confusing for OCR.

This chart represents the entirety of the ASD process from beginning to end.

Like any machine learning project, ASD begins at the input. The input is where the data is generated. Current throughput has established workflows for document generation to the Data Lake and unclassified STARS entries have been added to the Data Lake as well.

Future input will need to include the photographs as illustrated in this chart and later the films.

From the input, we move to Machine learning actual. There are two workflows embedded within ML both equally important: the development of training data and curating and standardizing of the data. The management of the data is critical to ensure sound outputs.

I have highlighted two various aspects of the Training data. Data cleaning requires an immense amount of effort in order to build to an outcome. Current efforts have a small data scraping team working to define basic characteristics of the documents.

However, future training workflows anticipate additional staff will be required to validate more data. For this reason, different types of data scraping teams will be required, some with backgrounds in nuclear science to bring out the nuances of the information and those with library science to ensure flexible frameworks are constructed. The same processes will be required for the photographs and films in the collection.

However, no machine learning software can read text or images directly so before curation can occur, XML parsing of documents and the stripping of images into layers will occur.

The later part of machine learning involves the operational effort of standardization.

It is at this point that the training elements developed in the first machine learning phase will be built out into models. As the models generate outputs, the outputs will be verified by staff prior to integrating the results in the output. Once common ground is laid for the framework, it is anticipated that some of these easier tasks will be automated thus leaving the more difficult problems for staff to work.

So what is our output? New STARS entries, increase of discoverability, measuring out of human versus machine problems, and then of course because machine learning is an iterative process, additional input for the initial stages of ASD.

ASD is a force multiplier not a people replacer. It is a way to expedite the information organization and retrieve‐ability of the DTRIAC collection. With good management and refinement, it takes the easy problems and queues them for the machine, and permits people to solve the harder problems.