Attachment J-5 MUSV Test Plan.pdf

Attachment J-5 Test Plan for

Medium Unmanned Surface Vessel

March 2026

1.0 INTRODUCTION

1.1 Purpose

This document establishes the objectives and measures of success for the Test and Evaluation (T&E) of the surrogate MUSV System for the MUSV 10 U.S.C. 4022 prototyping. It provides a framework for verifying system performance against attributes, evaluating operational effectiveness, and assessing overall system maturity. MUSV prototype testing consists of the following:

• Perception and Situation Awareness (PSA)

• COLREGS compliance & Vessel Safety

• Long-duration mission capabilities

1.2 Scope

This Test Plan addresses all test criteria, methodologies, scenarios, and data collection requirements for MUSV prototype testing. Vendors will develop the final test plan with specific test areas, planned order of testing, safety process/procedures, necessary regulatory approvals to conduct testing, testing contingencies, etc. The vendor test plan will incorporate all parts of this document. This document should be amended with the vendor specific testing information.

1.3 Document Overview

This document is organized into the following sections:

Section 1: Introduction and Scope Section 2: System and Test Asset Description and Limitations Section 3: Test and Evaluation Overview Section 4: Test Schedule and Duration Section 5: Test Procedures and Vignettes Section 6: Data Collection and Analysis Section 7: Roles and Responsibilities

2.0 SYSTEM AND TEST ASSET DESCRIPTION AND LIMITATIONS

2.1 USV Autonomy System (Test Article)

Description: A brief overview of the autonomy system architecture, including key software and hardware components (e.g., Autonomy Control System, Perception Suite, Machinery Control System).

Configuration Management (CM): The CM and Software versions tested shall be documented during the test. Additionally, the update process shall be defined and agreed to prior to testing.

Key Capabilities: Autonomous navigation, contact tracking, obstacle avoidance, COLREGS reasoning, mission planning, and remote monitoring.

2.2 USV Test Vessel (Surrogate or Representative Prototype)

Vessel Name/Class: [To Be Specified By Vendor] Length Overall (LOA): [To Be Specified By Vendor] Max Speed / Propulsion: [To Be Specified By Vendor] Sensor Suite: List of installed sensors (Radar, EO/IR cameras, AIS, etc.). [To Be Specified By Vendor] Placemat: [To Be Specified By Vendor]

2.3 Contact Vessels

Small Contact Vessel: Description (e.g., 7-12m LOA, speed capabilities, IMO number) [To Be Specified By Vendor] Medium Contact Vessel: Description (e.g., 15-50m LOA, speed capabilities, IMO number) [To Be Specified By Vendor] Chase / Safety Boat: Description of the vessel responsible for test monitoring and safety (If required) [To Be Specified By Vendor] Static Object: Description of the statis object used during 5.2.2 Test: Navigational Safety.

Static object shall be similar in size to a navigational buoy [To Be Specified By Vendor]

2.4 Go/No Go Criteria

Sea State Limitations: [To Be Specified By Vendor] Weather Limitations: [To Be Specified By Vendor] Visibility Limitations: [To Be Specified By Vendor] Equipment Limitations: [To Be Specified By Vendor]

Staffing Limitations: [To Be Specified By Vendor]

3.0 TEST AND EVALUATION OVERVIEW

3.1 General Test Approach

Testing will be conducted, organized into three primary phases: PSA Testing, COLREGS & Safety Scenarios, and Long Duration Mission. Tests will be conducted in a designated offshore test area and will involve a combination of live contact vessels and simulated data inputs. The Contractor shall be responsible for the test in its entirety including contacting local authorities to get approval for all test events.

3.2 Vessel Conduct and Safety

During the duration of operations, it is expected that the test vessel will operate in a safe and predictable manner. Successful performance criteria are presented in the individual test sections that follow. Throughout the duration of operations if any of the following occur that phase of testing will be considered unsuccessful:

• USV contact with a charted object.

• USV contact with not charted static objects excluding free floating objects deemed not of a size to cause damage to the USV (such as driftwood).

• USV collision or allision with another vessel with the exception of docking operations and passenger transfers between the USV and a support platform.

Failure of MCS to appropriately control machinery resulting in an unsafe condition of an HM&E component.

If a vessel violates the Closest Point of Approach (CPA) during phase 2 or phase 3 testing, then that will be deemed as an unsafe maneuver and will cause an automatic failure of that given phase. Operator intervention aboard the test platform to avoid violating CPA during testing will also be considered an automatic failure for phase 2 and phase 3. CPA for testing is listed in Table 1.

Table 1: Closest Point of Approach

Hazzard CPA (m) Small Vessel (LOA ≤12m)

To be provided at a later time

Medium Vessel (12m ≤ LOA ≤ 50m) Large Vessels (50m ≤ LOA) Vessel Not Under Command Charted shallow water (Water Depth ≤ 2*Draft) Charted Navigational Aids Uncharted Navigational Aids Charted Navigational Hazards

4.0 TEST SCHEDULE AND ESTIMATED DURATION

Table 2 shows the three test phases. Each phase is independently assessed with specific test criteria to satisfy. If any criteria are not met a for a given phase, then it is considered a failure. If a failure occurs that phase may be repeated to attempt to satisfy all criteria. Each phase may only be attempted twice to be successful. All phases must be satisfactorily executed and all criteria met to be considered successful for testing.

Table 2: Test Phases

Test Phase Planned Duration*

Phase 1: Perception & Situation Awareness

(PSA)

2 Days

Phase 2: COLREGS & Safety Vignettes 3 Days Phase 3: Long Duration Mission 1 Days Total Estimated Test Duration 6 Days

*This is an estimate, actual on water test time may vary

During testing the delineation for day and night will be sunrise and sunset as defined by the United States Naval Observatory for the latitude and longitude of the test area. Day testing will occur between sunrise and sunset. Night testing will occur between sunset and sunrise.

Detailed test scheduling is the responsibility of the vendor, government provided durations are estimates for informational purposes only and may not represent all required durations for phases or test scenarios. Phase 2 run(s) may be in between Phase 1 run(s) to improve efficiency of test execution schedule and reduce total number of on water days. Phase 3 must be completed in its entirety in the order prescribed. All test runs and scenarios shall be declared to onsite government personnel before they commence.

5.0 TEST PROCEDURES AND VIGNETTES

All tested scenarios or mission plans in Phases 2 and 3 will be in autonomous mode meaning that no operator input from a C2 station on or off the vessel will be tolerated until the scenario or mission is completed unless operator input is identified as allowed in the Government provided test plan. The test plan shall document how control of the vessel will be reported to the Government real time. During Phase 2 and Phase 3 testing, off board C2 of the vessel shall always be from the same control station with the location and instance of the control station declared in the test plan. If transfer to a secondary control station is required, then the Government will be made aware of the transfer of C2 shall witness operations from the secondary C2 station.

The operator is responsible for ensuring safe operation of all craft during testing.

5.1 Phase 1: Perception & Situation Awareness (PSA) Testing

5.1.1 Test: Circle PSA Testing

Total Estimated Time: 2 Days and Nights

Objective: To quantify the performance of the USV's radar and EO/IR sensors for detection, tracking, and classification.

For all PSA testing if other marine traffic necessitates a deviation from the test scenarios for safe operation, the vendor shall note these deviations, inform onsite government witnesses of the deviation, and clearly document the deviations in the test report.

Selection of test location should attempt to minimize disruptions from marine traffic by not operating in common marine traffic lanes or in areas restricted by navigational hazards.

Description: USV and contact vessels navigate opposing polygonal routes at various speeds, radii, and times of day to generate a continuous stream of sensor data across all aspect angles. The test scenario depicted in Figure 1 and defined in Table 3 shall be followed as closely as possible.

Only the required small and medium sized test contact vessels will be assessed against the perception test criteria. Other marine traffic that perception systems identify during testing will not be assessed.

Procedure Steps:

1. Configure USV and contact vessels with pre-defined waypoint tables

2. Disable Obstacle Avoidance and AIS input to perception suite on the test USV

3. Data shall be collected separately for passive perception (minus AIS), full perception (minus AIS), and AIS only.

a. The minimum recording rate shall be 1 Hz

4. Execute test per Figure 1 in accordance with Table 3 by declaring the start of scenario to the test personnel

5. Record all PSA sensor data, ground truth data from contact vessels, and environmental conditions.

6. The data will be assessed against the criteria in 5.1.3 Data Analysis and Success Criteria – Phase 1 Tests

Figure 1: Circles PSA Test

Table 3: Circle PSA Test Matrix R un N um be r

D ay /N ig ht

L ig ht

C on di tio ns Speed (kts) Rotation Radius (nm)

D ur at io n (h r)

U

SV

M

ED

IU

O N

TA

C

T

SM

A

LL

N

TA

C T

N

TA

C

T

SM

N

TA

C T

N

TA

C

T

SM

N

TA

C T

PD1 DAY 90% RPM* 15 15 CCW CW CCW 1 3.5 1.5 3

PD2 DAY 90% RPM* 15 15 CW CW CCW 1 3.5 1.5 3

PD3 DAY 6 15 15 CCW CW CCW 1 3.5 1.5 3

PN1 NIGHT 90% RPM* 15 15 CCW CW CCW 1 3.5 1.5 3

PN2 NIGHT 90% RPM* 15 15 CW CW CCW 1 3.5 1.5 3

PN3 NIGHT 6 15 15 CCW CW CCW 1 3.5 1.5 3

PD4 DAY 6 15 15 CW CW CCW 1 3.5 1.5 3

PD5 DAY 90% RPM* 15 15 CW CCW CW 1 3.5 1.5 3

PN4 NIGHT 6 15 15 CW CW CCW 1 3.5 1.5 3

*All engines online at that RPM, intended to assess at sustained speed of vessel

5.1.2 Test: Square Wave Classification

Description: Contact vessels perform passes by the USV to evaluate USV perception sensor performance, ability to maintain a contract track, and size-based classification performance.

Procedure Steps:

1. USV maintains minimal forward speed or Dynamic Positioning, to maintain constant heading. USV disables AIS input to the perception suite.

2. Both contact vessels (one small and one medium) operate at 15 knots and conduct two 12 nm passes each (STBD-to-PORT, bow-to-stern). Following the run matrix in Table 4.

Table 4: Square Wave Test Matrix

Run Number Day/Night Conditions Small Contact Medium Contact SW1 Day Starboard to Port Stern to Bow SW2 Day Stern to Bow Starboard to Port SW3 Night Starboard to Port Stern to Bow SW4 Night Stern to Bow Starboard to Port

3. The square wave is conducted to the design of 1 nm per direction traveled before turning, keeping as sharp as possible turns for the contact vessels.

4. Green arrows below show the travel direction, from approaching STBD to leaving port, and approaching bow leaving stern.

5. Passing vessel speed shall be defined by the contractor.

6. Repeat tests in daylight and night conditions.

Figure 2: Square Wave PSA Test (image not to scale)

7. Contact vessels shall display applicable navigation lights in accordance with the 1972 COLREGs.

8. Data shall be collected separately for passive perception (minus AIS), full perception (minus AIS), and AIS only.

9. Report Results in accordance with Section 5.1.3 and CDRL E001

10. These tests will be assessed against the criteria in 5.1.3 Data Analysis and Success

Criteria – Phase 1 Tests

5.1.3 Data Analysis and Success Criteria – Phase 1 Tests

The following information shall be reported after the PSA testing is completed to document Perception and Situational Awareness System. Results shall be presented against the criteria in

6 nm

1 nm

6 nm

Table 5 and

Table 6. For all Phase 1 testing only the prescribed contacts shall be used in the analysis.

Vessels in the vicinity of the test that are not part of the PSA test scenarios should be excluded from analysis. All metrics are cumulative over all PSA test scenarios.

Figure 3: PSA Relative Bearing Sectors for Assessment and Reporting (circles in 1 nm increments)

The

Table 6 track probability criteria are the threshold for acceptance of a PSA system. All criteria for each sector are expected to be met in order for a system to be deemed compliant with MUSV metrics. For the closest distance range the minimum value is intended to account for ship blockage near the vessel due to angle of view of sensors. If this blockage distance is greater than 0.1 nmi then that shall be reported to the government ahead of any testing events.

The following equation provides the calculation of Track Probability assessed against the

Table 6 criteria.

𝑻𝒓𝒂𝒄𝒌 𝑷𝒓𝒐𝒃𝒂𝒃𝒊𝒍𝒊𝒕𝒚 = 𝑇𝑖𝑚𝑒 𝐶𝑜𝑛𝑡𝑎𝑐𝑡 𝐷𝑒𝑡𝑒𝑐𝑡𝑒𝑑𝑇𝑜𝑡𝑎𝑙 𝑇𝑖𝑚𝑒 𝑜𝑓 𝑤𝑖𝑡ℎ𝑖𝑛 𝑃𝑟𝑒𝑠𝑠𝑐𝑟𝑖𝑏𝑒𝑑 𝑍𝑜𝑛𝑒 × 100

Table 5: Track Probability Full Perception Suite

Distance Zone Small Contact Medium Contact

0.1 to 1 nm

A

To be provided at a later time

B C D E F

1 to 2 nm

G H I J K L

2 to 3 nm

M N O P Q R

3 to 4 nm

S T U V W X

Table 6: Track Probability Criteria of Passive Perception

0.1 to 1 nm

A

To be provided at a later time

B C D E F

1 to 2 nm

G H I J K L

2 to 3 nm

M N O P Q R

3 to 4 nm

S T U V

5.2 Phase 2: COLREGS & Safety Scenarios

It is recommended that the test scenarios be scheduled to optimize efficiency. For example, overtaking scenarios are recommended to be followed up with the vessel being overtaken scenario. Scenarios are not required to be executed in order. The schedule for planned execution of all scenarios shall be provided in the test plan. Prior to commencing a scenario, the specific scenario will be announced to any Government witnesses and recorded in a test log.

Objective: To evaluate the autonomy system's ability to correctly interpret and execute maneuvers in accordance with COLREGs and navigate safely.

All test COLREGS and safety test scenarios will maintain a Closest Point of Approach (CPA) as defined in Table 1. If this CPA is violated at any point with the test contacts then that results in an automatic failure of Phase 2.

5.2.1 Test: Basic COLREGS Encounters

Head On:

The test vessel executes a head on course with contact vessels. Each vessel maintains the defined speed in Table 7. The contact vessel will not alter course.

Figure 4: COLREGS - Head On

Crossing Give Way:

The test vessel executes a crossing course with a contact vessel off starboard bow. For these scenarios a relative bearing of 90 degrees and 45 degrees will be assessed as shown in Figure 5 and Figure 6. Each vessel starts at the defined speed in Table 7 and the contact vessel shall maintain the prescribed speed for the duration of the test. The contact vessel will not alter course.

Figure 5: COLREGS - Crossing Give Way 90 Degree Relative Bearing

Figure 6: COLREGS - Crossing Give Way 45 Degree Relative Bearing

Crossing Stand-On:

The test vessel executes a crossing course with a contact vessel off port bow. For these scenarios a relative bearing of 270 degrees and 315 degrees will be assessed as shown in Figure 7 and Figure 8. Each vessel starts at the defined speed in Table 7 and the contact vessel shall maintain the prescribed speed for the duration of the test. The contact vessel will not alter course until it is within 3 times the CPA defined in Table 1 then the vessel will pass behind the test vessel at 2 times CPA.

Figure 7: COLREGS - Crossing Stand-On 270 Degree Relative Bearing

Figure 8: COLREGS – Crossing Stand-On 315 Degree Relative Bearing

Overtaking:

The test vessel approaches the contact from behind on the same heading, as shown in Figure 9 such that continuing the course would result in collision. Each vessel maintains the defined speed in Table 7. The contact vessel will not alter course.

Figure 9: COLREGS - Overtaking

Contact Overtaking:

The contact vessel approaches the USV from behind on the same heading, as shown in Figure 10. Each vessel maintains the defined speed in Table 7. The contact vessel will alter course to starboard when at a point 3 times CPA aft of the USV and maintain a 2 times CPA separation from the USV during the overtaking action.

Figure 10: COLREGs – Being Overtaken

Overtaking, Rule 13(d):

The test vessel approaches the contact from behind on an overtaking route that will cross the path of the contact, as shown in Figure 11. Each vessel maintains the defined speed in Table 7. The contact vessel will not alter course.

Figure 11: COLREGs – Overtaking Rule 13(d)

Contact Non-Compliance:

The test vessel executes a crossing course with a contact vessel off port bow identical in setup to a Crossing Stand on Scenario pictured in Figure 7 and Figure 8. Each vessel starts at the defined speed in Table 7 and the contact vessel shall maintain the prescribed speed and heading for the duration of the test.

Table 7: COLREG Runs

Scenario Run Number

Contact Bearing

(degrees relative)

Contact Speed

(kts)

USV

Speed

(kts)

Contact Vessel

Size

Sensor Source

Light Condition

(Day/ Night)

X Distance

Y Distance

Runs

Estimated Time Per Run (Hrs)

Estimated Time All

Runs (Hrs)

Head On

H1 0 16 16 Small Full Day 2 0 2 0.14 0.3 H2 0 16 16 Small Full Night 2 0 2 0.14 0.3 H3 0 16 16 Medium Full Day 4 0 2 0.28 0.6 H4 0 16 16 Medium Full Night 4 0 2 0.28 0.6 H5 0 16 16 Medium Passive Day 4 0 1 0.28 0.3 H6 0 16 16 Medium Passive Night 4 0 1 0.28 0.3

Crossing Give Way

CG1 270 16 16 Small Full Day 2 2 2 0.28 0.6 CG2 270 16 16 Small Full Night 2 2 2 0.28 0.6 CG3 270 16 16 Medium Full Day 4 4 2 0.55 1.1 CG4 270 16 16 Medium Full Night 4 4 2 0.55 1.1 CG5 270 16 16 Medium Passive Day 4 4 1 0.55 0.6 CG6 270 16 16 Medium Passive Night 4 4 1 0.55 0.6 CG7 315 17 12 Small Full Day 2 2 2 0.37 0.7 CG8 315 17 12 Small Full Night 2 2 2 0.37 0.7 CG9 315 17 12 Medium Full Day 4 4 2 0.73 1.5

CG10 315 17 12 Medium Full Night 4 4 2 0.73 1.5 CG11 315 17 12 Medium Passive Day 4 4 1 0.73 0.7 CG12 315 17 12 Medium Passive Night 4 4 1 0.73 0.7

Crossing Stand On

CS1 45 17 12 Small Full Day 2 2 2 0.37 0.7 CS2 45 17 12 Small Full Night 2 2 2 0.37 0.7 CS3 45 17 12 Medium Full Day 4 4 2 0.73 1.5 CS4 45 17 12 Medium Full Night 4 4 2 0.73 1.5 CS5 45 17 12 Medium Passive Day 4 4 1 0.73 0.7 CS6 45 17 12 Medium Passive Night 4 4 1 0.73 0.7 CS7 90 16 16 Small Full Day 2 2 2 0.28 0.6 CS8 90 16 16 Small Full Night 2 2 2 0.28 0.6 CS9 90 16 16 Medium Full Day 4 4 2 0.55 1.1

CS10 90 16 16 Medium Full Night 4 4 2 0.55 1.1 CS11 90 16 16 Medium Passive Day 4 4 1 0.55 0.6 CS12 90 16 16 Medium Passive Night 4 4 1 0.55 0.6

Contact Overtaking

CO1 180 16 7 Small Full Day 3 0 2 0.73 1.5 CO2 180 16 7 Small Full Night 3 0 2 0.73 1.5 CO3 180 16 7 Medium Full Day 3 0 2 0.73 1.5 CO4 180 16 7 Medium Full Night 3 0 2 0.73 1.5 CO5 180 16 7 Medium Passive Day 3 0 1 0.73 0.7 CO6 180 16 7 Medium Passive Night 3 0 1 0.73 0.7

Scenario Run Number

Contact Bearing

(degrees relative)

Contact Speed

(kts)

USV

Speed

(kts)

Contact Vessel

Size

Sensor Source

Light Condition

(Day/ Night)

X Distance

Y Distance

Runs

Estimated Time Per Run (Hrs)

Estimated Time All

Runs (Hrs)

USV

Overtaking

O1 0 7 16 Small Full Day 3 0 2 0.73 1.5 O2 0 7 16 Small Full Night 3 0 2 0.73 1.5 O3 0 7 16 Medium Full Day 3 0 2 0.73 1.5 O4 0 7 16 Medium Full Night 3 0 2 0.73 1.5 O5 0 7 16 Medium Passive Day 3 0 1 0.73 0.7 O6 0 7 16 Medium Passive Night 3 0 1 0.73 0.7

USV

Overtaking, Rule 13(d)

OC1 350 7 16 Small Full Day 3 0.95 2 0.75 1.5 OC2 350 7 16 Small Full Night 3 0.95 2 0.75 1.5 OC3 350 7 16 Medium Full Day 3 0.95 2 0.75 1.5 OC4 350 7 16 Medium Full Night 3 0.95 2 0.75 1.5 OC5 350 7 16 Medium Passive Day 3 0.95 1 0.75 0.8 OC6 350 7 16 Medium Passive Night 3 0.95 1 0.75 0.8

Contact Non-

Compliance Crossing Stand On

CN1 270 16 16 Medium Full Day 4 4 1 0.55 0.6 CN2 270 16 16 Medium Full Night 4 4 1 0.55 0.6 CN3 270 16 16 Medium Passive Day 4 4 1 0.55 0.6 CN4 270 16 16 Medium Passive Night 4 4 1 0.55 0.6 CN5 315 17 12 Medium Full Day 4 4 1 0.73 0.7 CN6 315 17 12 Medium Full Night 4 4 1 0.73 0.7 CN7 315 17 12 Small Passive Day 2 2 1 0.37 0.4 CN8 315 17 12 Small Passive Night 2 2 1 0.37 0.4

Note: Estimated Time listed for planning purposes only. Full suite is all sensors except for AIS, Passive is all sensors except for AIS and emitters.

Procedure Steps (General):

1. Position USV and contact vessel(s) at pre-defined start points and speeds as defined by Table 7 and Figure 4 through Figure 11. The start of each run should be declared at the C2 location and on the test support asset or test vessel bridge to the government witnesses.

2. Obstacle Avoidance on the USV will be turned on for COLREGs testing. For all scenarios a real contact of the size specified in Table 7 will be utilized. Automatic Identification System (AIS) inputs to the perception system will be disabled for the duration of testing, but AIS data should be logged and reported.

3. Any changes made to the ACS system configuration during the testing shall be documented. Observations made during testing will be documented in the test log to record notable interactions and events. Sea State and weather conditions including visibility will be recorded with timestamps.

4. Data shall be collected separately for passive perception (minus AIS), full perception (minus AIS), and AIS only.

5. Data collected onboard the USV and contact craft will be used to evaluate the path planner logic. Results and Data shall be delivered in the CDRL E001 Report.

6. Record all PSA sensor data, ground truth data from contact vessels, and environmental conditions. In accordance with 5.2.3 Data Analysis and CDRL E001.

7. Results of the test will be assessed against the criteria in 5.2.3 Data Analysis

5.2.2 Test: Navigational Safety

Test Scenarios: Detected Static Object, Keep In Zone, Keep Out Zone, Detected Shallow Water (with/without charts).

Static Object:

The test vessel approaches the static object on a course that it will cause it to pass within 1x USV LOA of the static object. The USV maintains the defined speed in Table 8. An acceptable USV response is to alter course to port or starboard maintaining the appropriate CPA while passing the object.

Figure 12: Safety Detected Static Object

Keep In Zone:

The test vessel is within a keep in zone that looks like a box within a box. The USV maintains the defined speed in Table 8. The vessel must demonstrate 2 required turns per run in this scenario. An acceptable USV response is to abide by the keep in zones and navigate the non-direct path from WP-0 to WP-1.

Figure 13: Safety – Keep in Zone

Keep Out Zone:

The test is on a course from WP-0 to WP-1 with a keep out zone that extends >5x CPA and blocks the USVs path. The USV maintains the defined speed in Table 8. An acceptable USV response is to abide by the keep out zones and navigate the non-direct path from WP-0 to

WP-1.

Figure 14: Safety – Keep Out Zone

Detected Shallow Water:

The test is on a course from WP-0 to WP-2 via WP-1. Detected shallow water/land that is less than navigable depth is in the straight line path from WP-1 to WP2 and extends from the boundary of the keep in zone on port to >5 USV lengths to starboard. Water depth at WP-0 and WP-2 is >10x minimum navigable depth. Water depth at WP-1 is ~2x minimum navigable depth. The USV maintains the defined speed in Table 8. An acceptable USV response is to capture WP-1 and then alter course to maintain adequate navigable depth below the hull and the capture WP-2. Water depth data input from the fathometer may be simulated to conduct this scenario.

Figure 15: Safety – Detected Shallow Water without Chart Data

Charted Shallow Water:

The test is on a course from WP-0 to WP-2 via WP-1. Charted shallow water/land that is less than navigable depth is in the straight line path from WP-1 to WP-2 and requires the USV to backtrack in order to avoid the shallow water. The USV maintains the defined speed in Table 8. An acceptable USV response is to capture WP-1 and then alter course to maintain adequate navigable depth below the hull and the capture WP-2.

Figure 16: Safety – Detected Shallow Water with Chart Data (for example purposes only)

Procedure Steps (General):

1. Obstacle Avoidance on the USV will be turned on for OA/COLREGs & Safety testing.

Automatic Identification System (AIS) inputs to the fusion will be disabled for the duration of testing, but AIS data should be logged to data analysis purposes.

a. Contacts will only be commercial traffic in safety testing if encountered during these scenarios.

2. Any changes made to the ACS system configuration during the testing shall be documented. Observations made during testing will be documented in the test log to record notable interactions and events. Sea State and weather conditions including visibility will be recorded with timestamps.

3. Data shall be collected separately for passive perception (minus AIS), full perception (minus AIS), and AIS only.

4. Data collected onboard the USV and contact craft will be used to evaluate the path planner logic. Results and Data shall be delivered in the CDRL E001 Report.

5. Set up the USV and the specified environmental hazard (e.g., virtual keep-out zone, charted shallow area). Based on the run in Table 8

6. Verify the USV's behavior adheres to the constraint (e.g., stays out of the zone, plots a course around the shallows).

7. Record all PSA sensor data, ground truth data from contact vessels, and environmental conditions. In accordance with Section 5.2.3 and CDRL E001.

Table 8: Safety Obstacle Avoidance Runs

Scenario Run Number Contact Bearing (degrees relative)

Contact Speed (kts)

USV Speed (kts)

Sensor Source

Light Condition (Day/Night) # Runs Time (Hrs)

Safety Detected

Static Object

ST1 0 0 16 Full Day 2 0.5

ST2 0 0 16 Full Night 2 0.5

ST3 0 0 16 Passive Day 2 0.5

ST4 0 0 16 Passive Night 2 0.5

Safety Detected Shallow

Water Not Charted

SWD1

0 n/a 16 n/a Day or Night 2 3.0

Safety Detected Shallow Water

Charted

SWC1

0 n/a 16 n/a Day or Night 2 3.0

Safety Keep In Zone

KI1 N/A n/a 12 n/a Day or Night 2 3.0

Safety Keep Out Zone

KO1 N/A n/a 12 n/a Day or Night 2 3.0

Note: AIS input not allowed for any sensor source

5.2.3 Data Analysis

For Phase 2 COLREGS scenarios executed in 5.2.1 Test: Basic COLREGS Encounters each run will be assessed for a successful USV response to the surface contact. Acceptable and unacceptable responses for the USV are defined in Table 9. The threshold for acceptable runs is shown in

Table 10.

Table 9: COLREG Behavior Summary

COLREG

Interaction

Acceptable Response Unacceptable Response

Head On Alter course to starboard Not altering course to starboard or failing to maintain CPA, reducing speed below 5 knots

Crossing Give Way Alter course to starboard and/or change speed to pass

Making initial turns to port, or reducing speed below 5 knots

Crossing Stand-On Maintain course and speed or make a slight turn to starboard

Reducing speed, turning to port or making an excessive turn to starboard >45 degrees from initial heading, reducing speed below 5 knots

Overtaking Alter course to starboard while passing the contact

Not altering course to starboard or failing to maintain CPA, reducing speed below 5 knots

Contact overtaking Maintain course and speed or slightly alter course to port to let the contact vessel pass

Altering course to starboard before contact has passed or significantly changing speed, reducing speed below 5 knots

Overtaking Rule 13(d)

Alter course to port ensuring the USV does not pass the contact by crossing its bow

Not altering course, violating CPA, or crossing the contacts bow, reducing speed below 5 knots

Non-Compliant Crossing Stand On

Alter course and/or adjust speed to let the contact pass

Crossing bow of contact at less than 1.5 times CPA, reducing speed below 5 knots

Table 10: COLREG Success Rate

Type Total Runs

Successful Runs

Success Rate (%)

CPA

violated

Head On 10

To be provided at a later date

Crossing, give way 20 Crossing, stand on 20

USV Overtaking 10 Contact Overtaking 10

USV Overtaking Rule 13d 8 Non-Compliant Crossing 8

For Phase 2 Safety Scenarios executed in 5.2.2 Test: Navigational Safety each run will be assessed for a successful USV response. Success for a run is measured by the USV having an acceptable response as defined in Table 11, ensuring safe operation of the vessel. All safety runs must be demonstrated successfully as summarized by

Table 12.

Table 11: Navigation Safety Summary

Test Acceptable Response Unacceptable Response Static Object Alter course to port or starboard while passing and maintaining

CPA

Not altering course or failing to maintain CPA

Keep In Zone Abide by the keep in zones and navigate the non-direct path

Deviating from keep in zones.

Keep Out Zone Abide by the keep out zones and navigate the non-direct path

Entering the keep out zone.

Detected shallow water Capture WP-1 then alter course to maintain adequate navigable depth below hull to capture WP-2

Not altering course to avoid the shallow water potentially grounding the vessel

Charted Shallow Water Capture WP-1, then alter course to maintain adequate navigable depth and capture WP-2

Not altering course to avoid the shallow water

Table 12: Safety Avoidance Statistics

Type Total Successful Success Rate

Detected Static Object 8

To be provided at a later date

Charted Shallow Water 2 Not Charted Shallow Water 2

Keep In Zone 2 Keep Out Zone 2

5.3 Phase 3: Long Duration Mission

Total Estimated Time: 1 Day

Objective: To demonstrate the system's ability to execute a multi-day mission profile with a variety of tasks and simulated operating limitations.

5.3.1 Test: Long Duration Mission Execution

Description: The USV will execute a continuous, mission profile composed of 25 distinct legs, as detailed in the Table 13. This phase of testing must conduct all steps in order without breaks. Upon capturing a given waypoint the vessel will immediately proceed to the next waypoint.

Procedure Steps:

1. For all scenarios a real contact of the size specified in the table will be utilized.

Automatic Identification System (AIS) inputs to the perception suite will be disabled for the duration of testing, but AIS data should be logged to data analysis purposes.

a. Waypoints may be commanded at 2 steps during testing for steps 1 through step 23, to start and at another step where comms are available. Steps 24 and 25 should performed with one step commanded at a time

b. Operator input outside of these specified times to command the autonomy suite will be considered a failure of phase 3 testing.

2. Table 13 specifies the status of obstacle avoidance, Radar, Cameras and Communications software and equipment.

3. Table 13 also defines the transit type for each segment. Transit types are defined as follows:

a. Waypoint Route: Navigating using specified latitude and longitude points with an identified capture radius

b. Station Keep: USV achieves and maintains assigned true or relative bearing and range from a designated contact

c. Vector: Navigating using and assigned true or magnetic course and speed.

USV can be program to (OA on) or not to (OA off) take action to prevent a collision with contacts.

4. Any changes made to the ACS system configuration during the testing shall be documented. Observations made during testing will be documented in the test log to record notable interactions and events. Sea State and weather conditions including visibility will be recorded with timestamps.

5. Data shall be collected separately for passive perception (minus AIS), full perception (minus AIS), and AIS only.

6. Data collected onboard the USV and contact craft will be used to evaluate the path planner logic. Results and Data shall be delivered in the CDRL E001 Report.

7. Load the mission plan into the USV's autonomy system. As listed in Table 13. For waypoint routes the capture radius should be set to 50 meters.

8. Initiate the mission.

9. Monitor the USV's progress as it automatically sequences through each leg of the plan.

10. During the mission, test personnel will trigger scripted "Hazard Interactions" (e.g., Head On, Overtaken) and system states (e.g., turning RADAR/Cameras/Comms on/off) at the times specified in the plan.

11. Record all PSA sensor data, ground truth data from contact vessels, and environmental conditions. In accordance with Section 5.3.2 and CDRL E001.

12. Demonstrate all remaining behaviors identified in the Prototype Proposal not demonstrated otherwise during this test plan.

Figure 17: Long Duration Mission

Table 13: Long Duration Mission Hour Mission

Start End

Tr an si t T yp e

Tr av el d is ta nc e

Sp ee d

Du ra tio n

(h rs

Pe rc ep tio n Su ite

St at us

Co m m un ic at io ns

Es ta bl is he d

Hazard Interactions

W ay po in t/ ep or di na te

Y or di po in t

X Co or di or di

1 Z L 2 D T Waypoint 23.4 12 2.0 Full Yes Head on 2 D T 3 J T Waypoint 6.0 25 0.2 Full Yes Overtaking 3 J T 4 Z O Waypoint 16.8 25 0.7 Full Yes Being overtaken 4 Z O 5 Z Z Waypoint 11.0 25 0.4 Passive Only Yes Crossing, give way 5 Z Z 6 P W Waypoint 10.4 25 0.4 Passive Only No Crossing, stand on 6 P W 7 Y K Waypoint 15.0 25 0.6 Passive Only No crossing, stand on 7 Y K 8 X M Waypoint 2.2 25 0.1 Passive Only No Head on 8 X M 9 L G Waypoint 13.4 12 1.1 Passive Only No Head on 9 L G 10 F V Waypoint 16.2 12 1.3 Full No Overtaking

10 F V 11 W M Waypoint 19.2 6 3.2 Full Yes Vessel travels in a simulated 0.25nmi wide channel with static channel markers

11 W M 12 K B Waypoint 16.3 6 2.7 Passive Only Yes Vessel travels in a simulated 0.25 nm wide channel with static channel markers

12 K B 13 Z N Waypoint 19.2 12 1.6 Passive Only Yes Crossing, give way 13 Z N 14 A O Waypoint 25.0 12 2.1 Passive Only Yes Overtaken 14 A O 15 O H Waypoint 15.7 25 0.6 Passive Only Yes crossing, stand on 15 O H 16 C Z Waypoint 21.6 25 0.9 Passive Only No Head On 16 C Z 17 G Z Waypoint 4.0 25 0.2 Passive Only No Being overtaken 17 G Z 18 Z V Waypoint 19.4 25 0.8 Full No Being overtaken 18 Z V 19 B M Waypoint 25.6 12 2.1 Full No Head On 19 B M 20 F T Waypoint 8.1 12 0.7 Full No Head On

Start End

Tr an si t T yp e

Tr av el d is ta nc e

Sp ee d

Du ra tio n

(h rs

Pe rc ep tio n Su ite at us m m un ic at io ns

Es ta bl is he d

Hazard Interactions

W ay po in t/ ep or di or di po in t

X Co or di or di

20 F T 21 A S Waypoint 5.1 6 0.8 Full Yes 21 A S 22 C N Waypoint 5.4 25 0.2 Passive Only Yes crossing stand on 22 C N 23 K Z Waypoint 14.4 25 0.6 Passive Only No crossing stand on 23 K Z 24 X R Waypoint 15.3 25 0.6 Passive Only Yes Head On

24 X R 25 C W Station Keep 21.6 25 0.9 Passive Only Yes Station Keep Contact @ 1nmi with a relative bearing of 45 degrees

25 C W 26 J S Vector (OA Off) 8.1 12 0.7 Full Yes

Vector with OA off Note: AIS input to PSA should be disabled for the duration of testing

5.3.2 Data Analysis

Criteria:

• To be provided at a later date

6.0 DATA COLLECTION AND ANALYSIS

6.1 Data Requirements

The following raw data shall be provided in a machine readable format compatible for the use on a portable computer using windows 11 or higher in accordance with CDRL E001.

Analyzed data shall be provided in the test report (CDRL E001). Data required by test phase is listed in

Table 14.

Table 14: Data Required by Phase

Test Phase Data to be collected Notes

Phase 1

PSA sensor data:

-Passive Sensors (no AIS) -Full Suite (no AIS) -AIS only

AIS only used as reference

Contact Location GPS location, sample frequency minimum 1Hz

Environmental Conditions -Temperature -Humidity -Visibility -Winds -Sea State

If able report sea state using closest wave buoy and relate to NATO STANAG 4194 scale

USV Location GPS location, sample frequency minimum 1Hz

Phase 2

PSA sensor data:

-Passive Sensors (no AIS) -Full Suite (no AIS) -AIS only

AIS only used as reference

Contact /Static Object Location Data

GPS location, sample frequency minimum 1Hz

Environmental Conditions -Temperature -Humidity -Visibility -Winds -Sea State

If able report sea state using closest wave buoy and relate to NATO STANAG 4194 scale

USV Location GPS location, sample frequency minimum 1Hz

Phase 1

PSA sensor data:

-Passive Sensors (no AIS) -Full Suite (no AIS) -AIS only

AIS only used as reference

Contact Location GPS location, sample frequency minimum 1Hz

Environmental Conditions -Temperature -Humidity -Visibility -Winds -Sea State

If able report sea state using closest wave buoy and relate to NATO STANAG 4194 scale

USV Location GPS location, sample frequency minimum 1Hz

During Testing the Test Director and C2 Operator shall take time based logs of events.

Logs should include: When underway, Start/Stop Test runs, Interventions required. A template of a Test log is provided in Table 15.

Table 15: Sequential Test Log Example

Time Test Phase Test Run Event

Table 16 shall be provided that documents the operator interventions for the systems not under autonomy tests in Phase 1, Phase 2, or Phase 3

Table 16: Human Intervention Summary

C2 Station Interventions Phase 1 Phase 2 Phase 3 QTY

Pilotage Perception

PNT

Vessel Control

SA

Total

Local Intervention required Phase 1 Phase 2 Phase 3 QTY

Autonomy/Perception Hardware Autonomy/Perception Software Lack of Remote SA Mechanical Failure Total

6.2 Analysis Procedures

Required analysis and acceptable criteria is listed in the individual sections. In addition, the following information shall be reported for the duration of the test.

Contact recognition is assessed by Table 17. This metric assesses the ability of the camera vision system to correctly recognize the size and propulsion type of a contact. A successful classification would report if the contact is a powerboat or sailboat and also the size category of the contact. A Small contact would be less than 15m and a medium contact would be greater than 15m and less than 50m. The calculation for Probability of Recognition is below:

𝑷𝒓𝒐𝒃𝒂𝒃𝒊𝒍𝒊𝒕𝒚 𝑹𝒆𝒄𝒐𝒈𝒏𝒊𝒕𝒊𝒐𝒏 = 𝑆𝑢𝑐𝑐𝑒𝑠𝑠𝑓𝑢𝑙 𝐶𝑙𝑎𝑠𝑠𝑖𝑓𝑖𝑐𝑎𝑡𝑖𝑜𝑛𝑠𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝐴𝐼𝑆 𝑉𝑒𝑠𝑠𝑒𝑙 𝐸𝑛𝑐𝑜𝑢𝑛𝑡𝑒𝑟𝑠 × 100

Table 17: Probability of Successful Classification of Contacts

0.1 to 1 nm

A

To be provided at a later date

B C D E F

1 to 2 nm

G H I J K L

2 to 3 nm

M N O P Q R

3 to 4 nm

S T U V

7.0 ROLES AND RESPONSIBILITIES

Vendor Roles and Responsibilities

• [To be specified in test plan]