Attachment_1_-_DNDO_Replay_Requirements_Specification_January_2016.pdf

AOS Report No. (AOS-16-0082)

January 2016

DNDO REPLAY TOOL REQUIREMENTS

SPECIFICATION

Version 2.0

Prepared for: DNDO

Prepared by: Linda Moniz and Robert Feuerbach

Task No.: D2807

Contract No.: HSHQDC-13-C-00005

[Replay Requirements] 1/21/2016 Page ii

CONTENTS

Page

Tables ............................................................................................................................................. iii

Preface............................................................................................................................................ iv

1. Introduction and Purpose

2. Scope and Overview

2.1 Scope

2.2 Requirements Overview

2.3 Contextual Definitions

2.4 Abbreviations and Acronyms

3. Requirements

3.1 General Requirements

3.2 Detail-Level Requirements

4. Notes To Industry

5. References

Appendix A. Injection File Example

[Replay Requirements] 1/21/2016 Page iii

TABLES

Page

Table 1 Requirements Matrix

Table 2 Detailed Requirements by Device Type

Table 3 Validation and Verification Criteria

[Replay Requirements] 1/21/2016 Page iv

PREFACE

Since 2007, DNDO has used replay tools in multiple programs to regression-test and evaluate algorithm and settings changes. In Spring 2015, DNDO circulated a request for information on existing replay tool capabilities to nuclear detection device vendors [7,8]. The replay tool requirements in this document were derived both from DNDO’s experience with replay and injection analysis and from the responses from industry.

A verified replay tool will exactly reproduce the algorithm results produced by the software in a radiation detection system. The primary requirement for any replay tool is the exact reproduction of the analysis results from the device, given identical sensor data and parameter inputs. Utility of such a tool to the vendor includes the ability to change software settings and parameters and to fine-tune an algorithm. The injection capability of the replay tool also allows the vendor to test “what-if” scenarios for theoretical changes to the detector’s hardware and their effect on the analysis results.

Prior to 2016, acquisitions of radiation detection and identification systems by DNDO did not include a requirement for a replay tool. The specification of requirements for the replay tool in advance of an acquisition is intended to improve efficiency for both the vendor and DNDO.

By stating these requirements for a replay tool in this document, the intent for future acquisitions is to enable vendor development of replay capability, suitable for use by DNDO, concurrently with development of the device and its software. The existence of a functioning replay tool that fulfills these requirements allows DNDO to concentrate efforts on the thorough evaluation of the detection systems.

Requirements for the replay tool rely heavily on the American National Standards listed in the references. These standards include those for the radiation detection systems themselves, and for the data format used by the systems to record measurement and analysis results data.

[Replay Requirements] 1/21/2016 Page 1

1. INTRODUCTION AND PURPOSE

This document both introduces the concept of a required replay tool for radiation detection systems that are to be considered for acquisition by the Domestic Nuclear Detection Office

(DNDO), and provides requirements for that replay tool. The requirements include a core set of requirements that are applicable for a wide range of radiation detection systems as well as requirements more specifically targeted to particular types of detection systems.

A replay tool in this context is a stand-alone version of the algorithms employed in the radiation detection (and, when applicable, categorization/identification) system to generate alarm decisions and indicators. Its primary function is to reproduce the algorithm that is used in the system as closely as possible, given hardware differences between those in the detection system and those in the platform (e.g. x86 PC) on which the replay tool is installed.

The primary use of a replay tool is to investigate a system’s performance over a much larger parameter space and under more controlled conditions than is possible through direct experimentation [8]. Replay tools are used by DNDO during multiple phases of system acquisition, testing, piloting, and deployment.

2. SCOPE AND OVERVIEW

2.1 Scope

The concept of a replay tool and its utility is generally applicable to detection and identification systems where sensor data (e.g., chemical, radiological, electromagnetic) is processed via computational algorithms to generate alarm decisions and indicators which are then used by an operator to make operational decisions in the field. For this document nuclear radiation detection systems are considered, specifically those in the following device types: personal radiation detectors (PRDs), hand-held detection and identification instruments, radiation portal monitors (RPMs), spectroscopic portal monitors (SPRMs), and backpack devices. Other types of nuclear detection systems are not specifically addressed in this document but the general requirements for a replay tool, particularly for devices that record sensor data, are applicable.

2.2 Requirements Overview

As mentioned previously the replay tool requirements in this document were derived from experience with replay and injection analysis as well as from responses from industry. They can be loosely organized into nine categories: reproduction (of live results), autonomy, injection, output, input, batch processing, compatibility and portability, verification and validation, and documentation. The following is an overview of some of the motivation behind key requirements.

The overarching requirement for a replay tool is the reproduction of analysis results from the detection system’s in-device algorithm. The requirements for autonomy together with reproducibility, in particular independence of results with order of processing must be counterbalanced with the use of adaptive technology in a recognition algorithm in order to improve detection performance based on evolving conditions. For example, an intelligent sensor system may use recent data to update backgrounds, calibration, or sensor status. In order to

[Replay Requirements] 1/21/2016 Page 2 achieve reproducibility together with autonomy, the internal state of the recognition algorithm should be traceable (and reproducible) and any interdependencies noted as part of verification and validation.

One important use of replay tools by DNDO is adjustment of recognition algorithm parameters; for example, in the past replay analysis has been used to tune an algorithm to ignore certain types of naturally occurring radioactive materials (NORM) alarms. Thus any parameters that are tunable via controls in the detection system should be tunable in the replay tool. It is also recognized that the replay tools can be useful to the vendors to explore further capabilities of the system. Additional parameter settings (that are not available on the detection system) may be built into the replay tool by the vendor for this purpose. However, as the primary use of the replay tool to DNDO is to mimic the live output, any additional parameters or parameter ranges available in the replay tool must be clearly labelled as such to avoid inadvertent use.

DNDO must evaluate and compare detection systems within a device type without bias. A consistent output format is necessary for rigorous evaluation of the detection systems and their recognition algorithms. The outputs are required to be in standard N42.42 (2012) format [1] to facilitate these evaluations and comparisons. Detection systems within a device type are also required to include the same input data types in their respective input files. Synthetic data are often used to explore the systems’ performance over a large parameter space. Thus, the ability to accept standardized synthetic spectra as input is required to evaluate the systems fairly.

DNDO will typically replay multiple files to assess device capability. Batch or command line processing is essential to the usefulness of the replay tool. There is some flexibility in the requirements (e.g. a batch mode, virtual machine, or batch “wrapper”) regarding the method the vendor uses to facilitate batch processing.

The replay tool needs to be compatible with various commonly used computing platforms

(e.g. virtual machines, MS-Windows-based systems) in order to be utilized by DNDO and its agents. Each installation of the replay tool is required to be validated against the installations at the vendor’s location; the vendor maintains responsibility for providing adequate validation files.

Documentation of the replay tool is an essential requirement for the use of the replay tool.

Although it is necessary to document the replay algorithm so it can be run independently and without the vendor’s oversight, internal workings of the algorithm need only be documented sufficiently to understand inputs, parameter settings, and outputs in the context of the operation of the replay tool and the interpretation of its results.

It is expected that a recognition algorithm will be updated as testing progresses and technology improves. Similarly, it is expected that updates to the replay tool will be done in parallel to maintain reproducibility. Thus, labeling of software, hardware, and firmware versions is essential in output files.

2.3 Contextual Definitions

Analysis

Parameters

Default or user-supplied information used by the radiation recognition algorithm that governs the algorithm’s processing of sensor data and ancillary data.

[Replay Requirements] 1/21/2016 Page 3

Ancillary Data (Digital) data provided by the detector that do not include the sensor’s foreground radiation measurements but is used by the radiation recognition algorithm. Ancillary data include calibration, background measurements, and other data necessary for processing with the algorithm but do not include analysis parameters.

ANSI N42.42 American National Standard Data format AN42.42 (2012) [1] for Radiation

Detectors used for Homeland Security

Background

Data

Sensor data marked by the detection system as being collected in the absence of any potentially hazardous environment, object or vehicle to be inspected;

data from the ambient radiological environment

Batch

Processing

A facility for processing multiple input files with a single launch statement or action; in this context refers to either a batch processing mode or a command line interface

Detector The component of a detection system that uses a sensor or sensors to measure gamma or neutron radiation and provides the sensor data to the radiation recognition algorithm

Detection

System

Gamma and/or neutron detector(s) coupled with the radiation recognition algorithm and user interface. The detection system provides alarm decisions and/or identification notifications to the operator.

Device Type In this document, a personal radiation detector, a handheld detection and identification device, a radiation portal monitor, a spectroscopic radiation portal monitor, or a backpack detection and identification device.

Foreground

Data

Sensor data from an inspection, to be differentiated from background data.

Injection The process by which synthetic data (from existing data and/or models) is used to alter an existing input file or to fabricate an entirely new synthetic input file, for processing by the replay tool.

Inspection The set of measurements collected and assessed for nuclear radiation by the detection system. In the case of an RPM or SRPM, the measurements are of a vehicle. Inspections by PRDs, Hand-held radiation detection and identification instruments, and backpack systems may be of the local radiation environment or of a targeted object.

Input Data The (raw) digital sensor data and ancillary data that are produced by a detector for input into the in-situ (i.e., in-device) algorithm or to the replay tool.

Input File A file that contains input data.

Live Describing algorithm output as calculated by the detection device’s radiation recognition algorithm as it is run in the device (“in-situ”).

Native File

Format

Data format determined by the vendor

[Replay Requirements] 1/21/2016 Page 4

Occupancy Data The input data for the duration of exactly one vehicle or object presence

Output File The file that results from the in-situ radiation recognition algorithm’s processing of data from a detector OR the file that results from a replay tool’s processing of an input file. It contains the analysis results, such as alarm decisions and/or nuclide identifications.

Radiation

Recognition

Algorithm

The algorithm used to detect, identify or categorize radiation signatures in the input data. The output (alarm, identification, and/or categorization) is dependent on both the data input to the algorithm and the algorithm’s processing. For example, a spectrum with sufficient resolution to identify photo peaks is necessary for identification. The terminology is shortened to

“recognition algorithm” in the text of the requirements.

Raw Data Unprocessed but digitized sensor data provided by the detector suitable for injection analysis

Replay Describing a recognition algorithm’s output as calculated by the replay tool

(vs. “live” calculated by the algorithm in the detection system). When used as a verb, the act of using the replay tool to process an input file or files.

Replay Tool A stand-alone software tool that mimics, given an input file, the signal processing, recognition algorithm, and alarm logic of a radiation detection system.

Results File Synonymous with Output File.

Standard

N42.42 Input

File

An input data file that is compliant with the N42.42 (2012) standard for a particular device type. Examples appear in the appendix.

Synthetic Data Sensor data that is not directly measured, but generated from modeling or from a combination of measured data and modeling.

Translator

Facility

A software program that changes an input file, if not human-readable and modifiable, to a human-readable and modifiable format and translates back to the original format without data loss.

Validation The process by which the installation of a replay tool installed on a platform other than the vendor’s is proven to yield results that are identical to those achieved by the installation on the vendor’s platform.

Verification The process by which a replay tool’s reproduction of the live detection system’s output is proven.

Wrapper

Software

The software that manages the processing of input data files by the radiation recognition algorithm, functioning as an interface between the algorithm and the operating system.

[Replay Requirements] 1/21/2016 Page 5

2.4 Abbreviations and Acronyms

DNDO Domestic Nuclear Detection Office

DLR Device Level Requirement

GR General Requirement

IP Intellectual Property

IV&V Independent Validation and Verification

JHU/APL The Johns Hopkins University Applied Physics Laboratory

NORM Naturally Occurring Radioactive Material

PRD Personal Radiation Detector

RPM Radiation Portal Monitor

U.S. United States

V&V Verification and Validation

3. REQUIREMENTS

3.1 General Requirements

The following table of general requirements refers to all device-types listed in the scope of this document. In some cases, the requirement must be modified for a particular device type. In this case the modification is listed in the testing procedure. In other cases, the meaning is specified by additional information related to device type, found in Table 2 and Table 3.

[Replay Requirements] 1/21/2016 Page 6

Table 1 Requirements Matrix

General Requirements

Reproducibility The replay tool exactly reproduces the device algorithm output when given identical input

REP Requirements pertaining to Reproducibility Proposed Tests

GR-1-REP

The replay tool shall exactly reproduce the live results of the radiation detection system under test when provided the identical input data

The meaning of “exactly reproduce” is device-type dependent as specified in Table 3 under “verification criteria.”

Compare the set of verification live output files to those output files obtained by replaying the verification input files.

A procedure, as indicated in Table 2, shall be documented to test this requirement for radiation detection systems that are not required by the ANSI standard to produce input data files.

Allowed differences are listed in Table 3.

Verify the replay tool by comparing results files from the live system and the replay tool:

• Alarms and identified isotope alarms, when applicable, shall be identical.

• Alarm and identification confidence, when applicable, shall be identical to the number of reported decimal places (dependent on device type).

• Identification results in output files (when applicable to device type) shall include identical sets of isotopes, even when the confidence of such isotopes does not merit an alarm.

• When applicable for device type, categorization (e.g.

industrial, medical, SNM) results shall be identical for identified isotopes and alarms.

• Measured data written in the output files, when applicable for device type, shall be identical.

• Pre-processed data written in the output files, when applicable for device type, shall be identical.

[Replay Requirements] 1/21/2016 Page 7

GR-2-REP The replay tool shall accept as its only required inputs the input data provided by the radiation detection system along with any analysis parameters used by the detection system.

Differences in the implementation of this requirement for radiation detection systems that do not record input data files are specified in Table 2. In the event that input data must be pre-processed before analysis by the recognition algorithm, the replay tool shall perform the pre-processing. This is in particular to enable the processing of synthetic data by the replay tool.

Note: The internal structure of the replay tool (i.e. the number of components) is not prescribed.

For radiation detection systems that produce input files, test that the replay tool will run and produce output files from the specified input data.

For radiation detection systems that do not produce files that can be input, verify that the properly formatted captured or simulated data can be replayed and produces the same alarm status as the live system under similar conditions as described in Table 3.

Test that the results from the replay tool match the results from the in-situ algorithm for all verification files, as for GR-1-REP.

GR-3-REP The replay tool shall have a user-accessible facility for adjustment of algorithm parameters.

User-adjustable parameters in the replay tool shall be set via parameter input files, and optionally a user-interface facility in the replay tool.

Test that parameters can be adjusted as documented.

GR-4-REP The replay tool shall include the same range and functionality for adjusting algorithm parameters as the in-situ algorithm on the radiation detection system.

Any parameters or parameter ranges in the replay tool not accessible to the operator on the radiation detection system shall be clearly indicated to the user as such before the replay is initiated.

Check that parameters that can be adjusted via settings on the radiation detection system can be set in the replay tool.

Check that setting any parameters that fall outside the parameter range accessible on the detection system causes a message, alarm, or other clearly identifiable notification before the replay of the file or files is initiated.

[Replay Requirements] 1/21/2016 Page 8

GR-5-REP Reproducibility of results shall be achievable as in

GR-1-REP independent of the order of processing.

Verify that the output (results) files are identical to the output validation files provided in the validation package when the input validation files are replayed in any order.

Autonomy The stand-alone tool has no additional software or run requirements

AUT General requirements pertaining to Autonomy Proposed Tests

GR-6-AUT The replay tool shall be a stand-alone software tool that can be installed and operated autonomously.

Autonomous operation in this context means without use of web services, an internet connection, or other devices.

Install the replay tool on each standalone (internet-disconnected) supported platform and check that the replay tool can replay all input validation files.

GR-7-AUT The replay tool shall process each input file separately.

Same test as GR-5-REP

[Replay Requirements] 1/21/2016 Page 9

Injection Injection and synthetic input file generation capability exists

INJ General requirements pertaining to Injection Proposed Tests

GR-8-INJ The replay tool package shall include a translator facility to generate N42.42 files from vendor format raw data files, and translate the N42.42 files back into the raw data files suitable for use with the replay tool.

The gamma-ray spectral data, neutron count data, and energy calibration data shall be stored in standard

N42.42 data elements; vendor-specific auxiliary data may be in permitted extensions to N42.42.

See Appendix A for an example of N42.42 spectral data elements.

The translator facility shall be subject to the same requirements as the replay tool regarding autonomy, batch processing, and documentation as the replay tool itself.

Check that the translator can produce readable N42.42 files from an original raw input data file.

Check that the translator can produce a raw input data file from the N42.42 file, referred to as the translated raw input data file.

Test that the replay outputs of the original raw input file and the translated raw input data file are identical.

GR-9-INJ The replay tool shall be capable of processing synthetic input data for injection studies in the same manner as measured input data.

Test that reply of synthetic input data can be executed and will produce an output file consistent with the measured object data for the same source.

[Replay Requirements] 1/21/2016 Page 10

Output Reply results facilitate consistent evaluation of results across devices

OUT General requirements pertaining to Output Proposed Tests

GR-10-OUT The replay tool shall provide output (results) in

N42.42 -2012 format with device-type appropriate detail.

Device-appropriate detail shall be as prescribed in

Table 2.

Test that all output files produced by replaying the set of validation input files are valid N42.42 files with detail as required in the detailed level requirements for the device type.

GR-11-OUT The replay tool shall provide, when device-type-appropriate as prescribed in Table 2, traceability

(achieved either by inclusion or by cross-reference or pointer) to raw data, ancillary data and analysis parameters in the output (results) file.

Verify that all output files produced by replaying the set of verification input files include (when required) raw data, ancillary data, and analysis parameters via cross-reference or inclusion

GR-12-OUT The replay tool shall provide, when device-type appropriate as prescribed in Table 2, version information for the firmware in the output file.

Verify that all output files produced by replaying the set of verification input files include (when required) the appropriately tagged version information for the firmware.

GR-13-OUT The replay tool shall provide, when device-type appropriate as prescribed in Table 2, hardware identification in the output (results) file. Hardware identification shall be via the appropriate XML tag from the N42.42 (2012) standard [1].

Check that all output files produced by replaying the set of verification input files include (when required) the appropriately tagged hardware identification.

GR-14-OUT Software version for the recognition algorithm used in the replay tool shall appear in the output (results) file for all input files processed by the replay tool.

Check that the software version for the replay tool’s detection algorithm is appropriately tagged and listed in all output files produced by replaying the input validation files.

GR-15-OUT Software version for the recognition algorithm used in the detection system shall appear in the output

(results) file for all output files produced by the in-situ recognition algorithm.

Check that the software version for the (in-situ) recognition algorithm is appropriately tagged and listed in all output files produced by the in-situ recognition algorithm.

[Replay Requirements] 1/21/2016 Page 11

Inputs Traceability of analysis parameters and input data from input to output.

INP General requirements pertaining to Input Proposed Tests

GR-16-INP Device-type appropriate input data as defined in

Table 2 shall appear in the input file.

Check that the files produced by the detection system and used as input to the replay tool include (when required) the appropriately tagged data.

GR-17-INP Model name and serial number for the radiation detection system shall appear in the input file, regardless of the format (native or ANSI N42.42-

2012) used.

Check that the files produced by the device and used as input to the replay tool include the appropriately tagged model name and serial number for the detection device. Native format files may be checked by verifying the model name and serial number is present in the translated data files.

GR-18-INP Hardware identifier and firmware (when applicable) version(s) for the detection system as specified by the device-type requirements in Table 2, should appear in the input file when possible.

Firmware includes any firmware used in the detection system to produce the input file.

Check that the files produced by the device and used as input to the replay tool include the appropriately tagged hardware identifiers and firmware version(s) for the detection system.

Check that these data are present in native format files after translation.

GR-19-INP Background and foreground data shall appear in the input file from the detection system according to the requirements listed in Table 2.

Check that the input files produced by the detection system and used as input to the replay tool include

(when required) the appropriately tagged background and foreground data.

Batch Mode

/Command Line

Efficient replay of large numbers of files

BAT General requirements pertaining to Batch Processing Proposed Tests

GR-20-BAT The replay tool shall provide a facility for batch processing of a minimum of 5,000 input files without further human intervention. The batch facility should be callable from a command-line interface.

The batch facility may be a wrapper around a replay tool program that processes a single file, or the primary replay tool facility may have this capability.

Check that a batch processing capability exists.

Check that at least 5,000 files can be processed without further human interaction after launch of the replay tool batch facility.

Check that parameters as designated in the tuning of the replay tool are listed in the output files when the files are processed via batch processing.

[Replay Requirements] 1/21/2016 Page 12

GR-21-BAT During batch processing, processing of valid input files shall continue without interruption regardless of the presence of any errant, invalid, incomplete, rejected, or otherwise non-replayable input files.

Non-replayable input files shall be skipped by the replay tool and processing of valid input files continue.

Verify that errant, valid, incomplete, or otherwise rejected files are skipped by the replay tool and processing of valid input files continues when the errant files are inserted in the beginning, the middle, and the end of the batch of files.

GR-22-BAT Batch processing in the replay tool shall be subject to all requirements regarding singly-replayed files.

Check that each file processed via batch mode corresponds to a uniquely labelled output file.

Verify that batch processing of input files produces identical output, with the exception of time stamp, regardless of the order in which the files are processed.

GR-23-BAT Designation of the input files for batch processing shall be either via a replay input directory under which all files will be processed, or via a file of input file names and path specifications to allow processing of files in a path other than that of the replay tool.

One or both methods may be implemented to satisfy this requirement.

A virtual-appliance-based replay tool should use an input-directory based approach.

If used, the file of filenames will be formatted as one path and filename per line.

Check that file names can be input to the command-line processing facility via a file of file names, including path, or by dropping the files into the input directory.

GR-24-BAT Designation of analysis parameters for batch processing shall be via a file of parameters (see GR-

3-REP) either specified at the start of the batch process, or introduced into the replay input directory.

Check that analysis parameters can be set by designating the path and filename of a parameter file, or by copying the parameter file into the replay input directory.

GR-25-BAT The batch processing facility shall provide a human-readable tabulated summary file from batch processing of files.

Check that the batch facility produces a file of tabulated data indicating correctly the number of files from a batch that are processed, skipped, rejected or

[Replay Requirements] 1/21/2016 Page 13

GR-26-BAT The batch facility shall flag the output of any errant file that can be processed by a unique identifier.

otherwise not processed, files with errors that are nonetheless processed, and the files that are alarming.

Check that the file of batch output is human-readable and independent of the order in which the files are processed.

GR-27-BAT The batch facility shall provide tabulated data from batch that indicates, for each batch, the number of files from the batch that are processed.

GR-28-BAT The batch facility shall provide tabulated data from batch that indicates, for each batch, the number of files from the batch that cannot be processed.

GR-29-BAT The batch facility shall provide tabulated data from batch that indicates, for each batch, the number of files that are errant but can be processed.

GR-30-BAT The batch facility shall provide tabulated data from batch that indicates, for each batch, the number of alarming and non-alarming files.

GR-31-BAT The batch facility shall generate a separate file of the names of non-replayable files from each batch formatted as one filename per line.

If the replay tool generates a reason or justification for the rejection of processing (e.g. “file truncated,”

“incomplete occupancy,” “unknown XML tag”), the batch facility should record that reason in the list of non-replayable files. Reasons or justifications may be inclusions or cross-references.

Insert non-replayable files into a batch processing list in the beginning, middle, and end of the list. Ensure in each instance that all non replayable file names appear in the separate output file’s list of non-replayable files and names of replayed files do not appear in this separate output file’s list.

If a cross-reference code for the rejection reason is recorded, check that a cross-reference is listed in the documentation for the replay tool.

[Replay Requirements] 1/21/2016 Page 14

Compatibility and

Portability

Backward data-handling compatibility; ability to run on standard computer platforms

COP General requirements pertaining to Compatibility and Portability Proposed Tests

GR-32-COP The replay tool shall be compatible with and provide installation mechanisms for standard-issue x86-based commercial computers running generally available standard-issue operating systems.

Such computers may be expected to run a supported version of Microsoft Windows or Linux at the time of the release.

Check that the replay tool can be installed and run on the designated platforms, and that results from each run of the validation files are identical to the extent required in Table 3.

GR-33-COP The replay installation package shall be complete, including all packages, libraries, third-party software, and licensing required to install and run the platform specific version of the replay tool.

If the replay tool requires the use of virtual machine or virtual appliance software, the software shall be included in the installation package and no additional licensing shall be required to run the replay tool.

Ensure that the installation package includes all required software for correct installation of the replay tool according to installation instructions.

GR-34-COP Updated versions of the replay tool shall be backward compatible with input files generated by earlier versions of the radiation detection system and processed by earlier versions of the replay tool.

Updated versions of the replay tool should be delivered, along with appropriate validation and verification data sets, whenever substantive changes are made to the radiation detection system’s algorithm.

Verify that the updated replay tool includes a change in the version number as written to the output file.

Test that subsequent versions of the replay tool are able to replay all files from the validation and verification data sets from previous versions.

Test that subsequent versions of the replay tool indicate the hardware and firmware version, with appropriate tag.

Test that any delivered version of the replay tool includes separate verification and validation packages of input and output files.

[Replay Requirements] 1/21/2016 Page 15

Validation and

Verification

Method and files to validate installation and verify reproducibility

VAV General requirements pertaining to Validation and Verification Proposed Tests

GR-35-VAV The replay tool shall provide validation files to ensure that the tool’s performance as described by the vendor is reproduced with each installation. The number of required installation validation files is listed in Table 3. Validation files include both the unanalyzed measurement input files and the vendor-produced replay output results files from processing the input files.

Check that the validation files are delivered with the replay tool. Verify that all validation input files can be run by the replay tool. Validate the installation and compare the results of the replay tool against the replays run by the vendor; differences allowed in replay results shall be limited to those listed in Table

3.

GR-36-VAV The vendor shall provide a minimum number of replay verification input files (unanalyzed input data files) and the corresponding verification output files generated by the detection system (“live”), with the number dependent on device type, appearing in Table

3.

These input files and others will be replayed to test

GR-1-REP.

Check that the correct number of verification input and output files are delivered with the replay tool. Verify that all verification input files can be processed by the replay tool.

[Replay Requirements] 1/21/2016 Page 16

Documentation Complete manual, file architecture, version documentation

DOC General requirements pertaining to Documentation Proposed Tests

GR-37-DOC The replay tool shall include a complete manual for installation, operation and troubleshooting.

• The replay tool shall include documentation for each parameter setting.

o Each parameter shall be identified and its effect on the recognition algorithm shall be indicated o Default parameter settings shall be identified in the documentation o If the label of a parameter in the in-situ algorithm in the device differs from the label of a parameter in the replay tool, this discrepancy shall be explicitly noted o The vendor need not reveal proprietary information

• The replay tool shall include definitions of statistical measures used in batch processing report.

Check that the manual is complete and comprehensive

Check that the required parameter documentation exists.

Check that definitions of the particular statistical measures used in batch or aggregate statistics exist and are among the standard definitions.

GR-38-DOC The replay tool documentation shall include the file input and output architecture description. This includes complete description of the data file formats and content.

Check that the documentation is delivered, complete, and matches collected data

[Replay Requirements] 1/21/2016 Page 17

3.2 Detail-Level Requirements

Table 2 Detailed Requirements by Device Type

Detailed Requirements by Device Type Personal Radiation

Detection Devices (PRD)

Depending upon the model, these radiation detection systems might not record or transmit measurements for replay. In this case, output of the system is limited to audible, vibrating, or visible alarms.

Verification and validation of the replay tool for this device type is expected to be less exacting than for a device that produces an input data file. Similarly, output from the replay device can be considerably simpler than that from other radiation detection and identification systems. The following are requirements that address those differences in input and output.

REP DLR-PRD-1-REP The replay tool shall take input data with the same fidelity provided to the in situ algorithm in the detection system along with any parameters provided to the recognition algorithm in the system. The

ANSI standard N42.33-2006 [1] requirement is that for any instrument that records data, the data format shall comply with ANSI N42.42-2012 [2].

REP DLR-PRD-2-REP For radiation detection systems that do not store and retrieve measurements, the spectral or other sensor information shall include the following in the documentation for the replay tools:

• Specification for how an input data file is generated and retrieved or transmitted for replay

• Specification for the generation of synthetic or injected data for replay.

INP DLR-PRD-3-INP For radiation detection systems that store and retrieve input data files, those files shall include at a minimum the following fields, either in native format or in compliant N42.42-2012 format.

Instrument identification

• Time, Date

• Detector Information (e.g. NaI)

• Hardware, software and firmware versions of the detector if input data files are written by the system

• Serial number or other unique identifier for the detection device if input data files are written by the system

• Unique occupancy or input data identifier

• Date/time of the input data

• When available, Dose Rate

• When available, Exposure Rate or Gross Counts

[Replay Requirements] 1/21/2016 Page 18

OUT DLR-PRD-4-OUT Standard replay output(results) files for personal radiation detection systems shall comply with ANSI

N42.42-2012 [2] format and include the following

• Alarm State (alarm/no alarm) for lights, audible alarm, and vibration alarm; and Alarm Description

• Detector material (e.g. NaI) used in the detector

• Software version for the replay

• Hardware, software and firmware versions of the detector if input data files are written by the system

• Serial number or other unique identifier for the detection device if input data files are written by the system

• Unique occupancy or input data identifier

• Date/time of the input data

• Date/time of the replay analysis

• When gross counts are available, gross counts

• When spectral data are available, the spectral data that generated the alarm decision

• When available, Dose Rate information

[Replay Requirements] 1/21/2016 Page 19

Handheld Radiation

Detection Instruments

(HH)

Handheld Radiation detection systems detect and identify radionuclides and indicate both gamma exposure rates and indications of neutron radiation. The ANSI standard N42.34-2006 [3] requires that the detection systems are able to internally store and transfer at least 50 complete unprocessed spectra.

Recorded measurement information shall also include time and date, spectrum integration time, measured gamma exposure time, and neutron count rate at the time of measurement. As stated in requirement GR-

11-OUT, the replay tool output file shall include this raw measurement data.

TRA DLR-HH-1-INP Handheld radiation detection instrument replay tool input (detector data) files shall include at a minimum the following fields, either in native format or in compliant N42.42-2012 format.

• Instrument identification

• Time, Date

• Detector Information (e.g. NaI)

• Energy Calibration

• Foreground

• Background

• Live time for each spectrum measurement

• Spectrum Channel Data for each detector

• Dose Rate if recorded by the instrument

• Gross Counts

• Definition of a measurement group that links background and foreground measurements

• Any Pre-processed data as it would be delivered to the in-situ algorithm

OUT DLR-HH-2-OUT Handheld radiation detection instrument replay tool output (results) files shall include the following fields, in ANSI n42.42-2012 compliant format.

• All Input data above in DLR-HH-1-INP

• Analysis Results (shall include alarm state)

• Analysis Start Date/Time

• Algorithm Component name and version for each component (e.g. detector, identifier)

• Nuclide information – name, confidence value, description (category, e.g. Industrial)

• Spectrum peak analysis results – peak energy value, peak net area value, peak net area uncertainty value

(if available)

[Replay Requirements] 1/21/2016 Page 20

Radiation Portal

Monitors (RPM)

Polyvinyl toluene (PVT) based RPM radiation detection systems typically detect or categorize but not identify radioactive materials. The ANSI standard N42.35-2006 requires that the devices are able to internally store and transfer at least 1000 complete occupancy data sets or, if occupancy sensors are not present, 3 hours of measurement data. Required information in the input data file includes time and date, occupancy or measurement time, monitor identification gamma count rate and neutron count rate [4] .

In addition the monitor shall be able to store gamma and neutron count rate time-history data. As stated in requirement GR-11-OUT, the replay tool output file shall include traceability to this raw input data.

INP DLR-RPM-1-INP Radiation portal monitor replay tool input files shall include the following fields, either in native file format or in compliant ANSI N42.42-2012 format.

• Portal Manufacturer

• Instrument ID

• Detector Type and Status for each detector

• Occupancy number (when applicable)

• Measurement location

• Conveyance speed (if applicable)

• Calibration Type, model, and coefficients

• For each time slice data record:

Start Time, Sample Real Time, and Live Time

Source Type (background, item, ...)

Spectrum or count data for each detector linked to calibration data

Occupancy presence code (if applicable)

OUT DLR-RPM-2-OUT Radiation portal monitor replay tool output (results) files shall include the following fields in ANSI

N42.42-2012 compliant format.

• Input data as above in DLR-RPM-1-INP

• Algorithm Version and Release Date

• Algorithm Configuration

• Alarm Data (Color, description/category if present)

• Analysis start time and end time

• Background Spectrum

• Gross Counts

• Alarm Confidence

• Radiation recognition algorithm metric (e.g., maximum gross-count sigma-above-background)

[Replay Requirements] 1/21/2016 Page 21

Spectroscopy-based

Radiation Portal

Monitors (SRPM)

Spectroscopic based radiation detection systems detect and identify radionuclides. The ANSI standard

N42.38-2006 requires that the systems are able to internally store and transfer at least 1000 complete occupancy data sets; the devices are also required to have both occupancy and speed sensors. Recorded measurement information includes the unprocessed spectrum or spectra for each occupancy, time and date, occupancy time, monitor identification, background count rate and spectrum, gamma count rate, neutron count rate, and gamma and neutron count rate time-history data [5]. As stated in requirement

GR-11-OUT, the replay tool output file shall include traceability to this raw measurement data.

INP DLR-SRPM-1-INP Spectroscopy-based radiation portal monitor replay tool input files shall include the following fields, either in native file format or in compliant ANSI n42.42-2012 format.

• Portal Manufacturer

• Instrument ID

• Detector Type and Status (e.g. “operational”, “OK”) for each detector

• Occupancy number

• Measurement location

• Conveyance speed

• Calibration Type, model, and coefficients

• For each detector and time slice:

Start Time, Sample Real Time, and Live Time

Source Type (background, item, ...)

Spectrum or count data for each detector linked to calibration data

Occupancy presence code

• Any pre-processed data as it would be provided to the in-situ algorithm.

OUT DLR-SRPM-2-

OUT

Spectroscopy-based radiation portal monitor replay tool output (results) files shall include the following fields in compliant ANSI n42.42-2012 format.

• Input data as above in DLR-SRPM-1-INP

• Algorithm Version and Release Date

• Algorithm Configuration

• Alarm Data (Color, description)

• Analysis start time and end time

• Background Spectrum

• Nuclide information – name, confidence value, description/category (e.g. Medical)

• Spectrum peak analysis results – peak energy value, peak net area value, peak net area uncertainty value

[Replay Requirements] 1/21/2016 Page 22

Backpack devices (BP) Backpack radiation detection systems detect gamma radiation and may include neutron detection. They may identify gamma emitting radionuclides. The ANSI standard N42.53-2013 requires that the devices are able to store up to 8 hours of data records internally. Recorded information includes date, time and time zone, manufacturer, model, version and serial number for the device, GPS location, background gamma count, background neutron count if equipped with neutron detector, measured gamma and neutron count rates, and operational status [6]. Each recorded spectrum shall have a time stamp, real time, live time, and calibration coefficients. As stated in requirement GR-11-OUT, the replay tool output file shall include this raw measurement data.

TRA DLR-BP-1-TRA Backpack replay tool input files shall include the following fields, either in native file format or in compliant ANSI n42.42-2012 format.

• Instrument identification

• Time, Date

• Detector Information (e.g. NaI, He-3)

• Energy Calibration

• Foreground spectrum for each detector

• Background spectrum for each detector

• Live time for each spectrum measurement

• FWHM Calibration, if available

• Dose Rate (if applicable)

• Definition of a measurement group that links background and foreground measurements

• Any Pre-processed data as it would be delivered to the in-situ algorithm

OUT DLR-BP-2-OUT Backpack replay tool output (results) files shall include the following fields in compliant ANSI N42.42-

2012 format.

• All Input data above in DLR-BP-1-INP

• Algorithm Version and Release Date

• Algorithm Configuration

• Alarm Data (Color, description)

• Analysis start time and end time

• Background Spectrum

• Nuclide information – name, confidence value, description/category( e.g. NORM) if applicable

• Spectrum peak analysis results – peak energy value, peak net area value, peak net area uncertainty value

[Replay Requirements] 1/21/2016 Page 23

Table 3 Validation and Verification Criteria

Detection

System Type

Minimum number of replay installation validation input and output files*

Minimum number of replay verification input and output files**

Installation Validation Criteria Verification Criteria

Personal

Radiation

Detection device

(e.g. pager)

If input and output files are produced.

If input and output files are not produced, the validation procedure shall be documented as above

If input and output files are produced.

If input and output files are not produced, the verification procedure shall be documented as above

Identical alarm status, dose rate and input data

Identical alarm status, dose rate

Handheld detection and identification instrument

60 1000 Identical output, with the exception of time stamp and replay device operating system information; identification confidence within precision given in the live output file.

Identical input, <nuclide> data confidence within precision given in the live output file

Radiation Portal

Monitor

60 1000 Identical output, with the exception of time stamp and replay device operating system information.

Identical input, <AnalysisResults>

[Replay Requirements] 1/21/2016 Page 24

Spectroscopic

Radiation Portal

Monitor

60 1000 Identical output, with the exception of time stamp and replay device operating system information; identification confidence within precision given in the live output file.

Identical input, <nuclide>, <SpectrumPeak> confidence within precision given in the live output file

Backpack Device 60 10 hours of data Identical output, with the exception of time stamp and replay device operating system information; identification confidence within precision given in the live output file

Identical input, <Nuclide>, <SpectrumPeak> (if applicable) confidence within precision given in the live output file

*The validation data set should include a range of alarming and nuclide identification results, to be able to verify the replay tool is functioning properly when used by the government.

**The verification data set is used to test and confirm the replay tool is appropriate to use for analyses examining the system’s sensitivity. As such, the verification data set must include a wide range of non-alarming and alarming radiological conditions. The intensity (i.e., gamma-ray or neutron rate) of incident radiation from sources used to generate the dataset should span from approximately ¼ the intensity where the probability of detection (true positive rate) first exceeds 10%, to approximately 4x the intensity when the probability of detection exceeds 95% in logarithmic steps of doubling intensity.

[Replay Requirements] 1/21/2016 Page 25

4. NOTES TO INDUSTRY

Often a vendor’s algorithm is proprietary information and represents considerable development investment. DNDO’s interest lies in the demonstration of performance standards when implementing a detection system. It has no interest in knowing vendors’ trade secrets, only in quantifying differences in performance that differentiate the systems. Independent validation and verification of the requirements for the replay tool and the devices will be undertaken to maintain objectivity in the selection of a detection system or systems.

In order to safeguard the vendor’s…