Attachment_B_-_Test_Plan_(final).pdf

Density Meter Performance Evaluation Test Plan

Introduction To assist with the Density Meter procurement, a performance evaluation will be conducted as part of the selection criteria with emphasis on inspection rate and accuracy. The test results do not replace any DID or SOW requirement; it is still the vendor’s responsibility to ensure the Acceptance Test Plan (ATP) verifies the density meter meets all SOW requirements upon contract award. This plan may be modified to adapt to any changes or unforeseen circumstances, however, the objective of the test will remain unchanged.

Objective Verify the System Under Test (SUT) has a high probability of meeting SOW requirements.

Test Material

• Blocking plates - shall be of carbon steel of one of the following designations (C1008, C1018, C45, A36) and be 6mm and 8mm thick.

• Contraband Simulate – Cocaine simulate, see Table 1.

Table 1. Contraband simulate information

Implementation Three test scenarios will be used to demonstrate the SUT’s capabilities. The first scenario, Figure 1, demonstrates the SUT can successfully inspect an area without a change in density, i.e. no false positives. The second and third scenarios, Figure 2, will demonstrate the SUT’s ability to detect a change in density, i.e. increase and decrease in density at a specified inspection rate1.

Each device will be tested at two different thicknesses of steel (6 mm and 8 mm) while maintaining an inspection rate2 of 2”/sec or 4”/sec, see Tables 2 & 3 for scoring criteria. Each device will undergo three trials, with fifty samples taken for each trial, totaling 150 samples for a complete test. Using Equation 1, the accuracy will be calculated for each trial.

1 Accuracy of Density Meters will maintain or increase as inspection rate decreases.

2 The inspection rate will vary; however, most samples obtained at the specified inspection rate (2”/sec or 4”/sec).

True Composition

Simulate Composition Simulate formula / notes

Cocaine 6.5 wt% H 5.4 wt% H 18.4 wt% polyvinylchloride C17H22O4NCl 60.1 wt% C 61.9 wt% C 8.8 wt% urea Density* = 0.75 g/cc 4.1 wt% N 4.1 wt% N 72.8 wt% cinnamic acid

18.8 wt% O 18.1 wt% O

10.4 wt% Cl 10.4 wt% Cl

* Average density; density of real cocaine depends on how it is packaged. Expected densities are in the range of 0.5 to 1.1 g/cc.

Drug Simulant density is between 0.6 g/cc & 0.83 g/cc & has the approximate dimensions of 6" x 6" x 2".

Inspection Rate Correct Result Pass/Fail/Rescan Count sample < 1.50 in/sec Yes Rescan No > 1.50 in/sec Yes Pass Yes < 2.25 in/sec No Fail Yes > 2.25 in/sec No Rescan No

Table 2: Inspection rate scoring criteria for 2”/sec inspection rate

Inspection Rate Correct Result Pass/Fail/Rescan Count sample < 3.50 in/sec Yes Rescan No > 3.50 in/sec Yes Pass Yes < 4.25 in/sec No Fail Yes > 4.25 in/sec No Rescan No

Table 3: Inspection rate scoring criteria for 4”/sec inspection rate

𝐴𝐴𝐴𝐴𝐴𝐴 =

∑𝑇𝑇𝑇𝑇 + ∑𝑇𝑇𝑇𝑇

∑𝑇𝑇𝑇𝑇 + ∑𝑇𝑇𝑇𝑇 + ∑𝐹𝐹𝑇𝑇 + ∑𝐹𝐹𝑇𝑇

Equation 1. Accuracy

Accuracy (ACC) = The degree to which the indications of a change in density are correct.

True Positive (TP) = Indication of a change in density when there is one.

True Negative (TN) = No indication of a change in density when there is not one.

False Positive (FP) = Indication of change in density when there is none, “false alarm”.

False Negative (FN) = No indication of a change in density when there is one; does not alarm over change in density.

Appendix A Density Meter Qualification Test Scenarios

Figure 1. False positive test scenario

Figure 2. Change in density detection test scenario (increase and decrease)

Blocking Material

<--------------------------------------------------28"------------------------------------------------->

Position 1 Position 2

Blocking Material Contraband Simulant

(Organic)

Position 1 Position 2

<--------------------------------------------------28"------------------------------------------------->

Position 3

Appendix B Density Meter Qualification Test Procedure Trial 1. Threshold Inspection Level

Trial 2. Threshold penetration level with a faster inspection rate (above Threshold Inspection Level)

Trial 3. Objective Inspection Level

1 Place 6 mm of steel on test fixture

1.1 Baseline the System Under Test (SUT) at position 1 <= 4 seconds

1.2 Start stopwatch and move the SUT, at an inspection rate greater than 2"/sec, to position 2 and return to position 1

No Alarm

1.3 Stop stopwatch at postion 1 <= 28 seconds

2 Place contraband simulant under steel

2.1 Baseline the SUT at position 1 <= 4 seconds

2.2 Start stopwatch & move SUT, at an inspection rate greater than 2"/sec, to position 2

SUT alarms over contraband simulant

2.3 Stop stopwatch at postion 2 <= 14 seconds

3 Baseline the SUT at Position 3 <= 4 seconds

3.1 Start stopwatch & move SUT, at an inspection rate greater than 2"/sec, to Position 1

SUT continually alarms when NOT over contraband simulant

3.2 Stop stopwatch once the SUT alarms <= 4 seconds

Detect Density Decrease

(See Figure 2) m m S te el

Detect Consistent Density

(See Figure 1)

Detect Density Increase

(See Figure 2)

SOW Threshold Depth of Reading Requirement at 2"/sec Inspection Rate

Expected Result Observed

Result Pass/FailTest Detail Step Step Description

4 Place 6 mm of steel on test fixture

4.1 Baseline the System Under Test (SUT) at position 1 <= 2 seconds

4.2 Start stopwatch and move the SUT, at an inspection rate greater than 4"/sec, to position 2 and return to position 1

No Alarm

4.3 Stop stopwatch at postion 1 <= 14 seconds

5 Place contraband simulant under steel

5.1 Baseline the SUT at position 1 <= 2 seconds

5.2 Start stopwatch & move SUT, at an inspection rate greater than 4"/sec, to position 2

SUT alarms over contraband simulant

5.3 Stop stopwatch at postion 2 <= 7 seconds

6 Baseline the SUT at Position 3 <= 2 seconds

6.1 Start stopwatch & move SUT, at an inspection rate greater than 4"/sec, to Position 1

SUT continually alarms when NOT over contraband simulant

6.2 Stop stopwatch once the SUT alarms <= 2 seconds

SOW Threshold Depth of Reading Requirement at 4"/sec Inspection Rate

Expected Result Observed

Result Pass/FailTest Detail Step Step Description

Detect Density Decrease

(See Figure 2) m m S te el

Detect Consistent Density

(See Figure 1)

Detect Density Increase

(See Figure 2)

7 Place 8 mm of steel on test fixture

7.1 Baseline the System Under Test (SUT) at position 1 <= 2 seconds

7.2 Start stopwatch and move the SUT, at an inspection rate greater than 4"/sec, to position 2 and return to position 1

No Alarm

7.3 Stop stopwatch at postion 1 <= 14 seconds

8 Place contraband simulant under steel

8.1 Baseline the SUT at position 1 <= 2 seconds

8.2 Start stopwatch & move SUT, at an inspection rate greater than 4"/sec, to position 2

SUT alarms over contraband simulant

8.3 Stop stopwatch at postion 2 <= 7 seconds

9 Baseline the SUT at Position 3 <= 2 seconds

9.1 Start stopwatch & move SUT, at an inspection rate greater than 4"/sec, to Position 1

SUT continually alarms when NOT over contraband simulant

9.2 Stop stopwatch once the SUT alarms <= 2 seconds

Expected Result

SOW Objective Depth of Reading Requirement at 4"/sec Inspection Rate

Test Detail Step Step Description

Detect Consistent Density

(See Figure 1)

Detect Density Increase

(See Figure 2)

Detect Density Decrease

(See Figure 2)

Observed Result Pass/Fail m m S te el