FA8650-17-S-1070-Atch5.pdf

Classification of Vibrating Systems (CLOVIS)

STATEMENT OF OBJECTIVES (SOO)

3 February 2017

Keywords:

Classification, Identification, Mechanical Systems, Vibration, Measurements, Algorithms

Background: LiDAR vibrometers are able to measure the surface displacement of mechanically operating machinery at the micro-meter level. This ability provides a non-contact means of assessing the operating condition of devices with rotating elements (e.g. engines) or devices that hum from magnetostriction (e.g. high power transformers) and is normally measured using contact accelerometers or commercial LiDAR vibrometers. However, commercial LiDAR vibrometers are short range and only function at ranges of meters to 10’s of meters for single point measurements. Commercial vibrometers in conjunction with accelerometers are used for quality control, diagnostics, as well as noise and vibration analysis. The Air Force is interested in utilizing long range LiDAR vibrometers for identification of targets. Preliminary classification and identification algorithm research utilizing accelerometer measurements and limited LiDAR measurements from a variety of vehicles has demonstrated that sufficient identification separability exists between vehicles of different types. However, any LiDAR operating at military useful ranges would be unable to make the stable point measurements that commercial systems achieve at short ranges or the accuracy of accelerometer contact measurements.

Airborne vibrometers will be susceptible to issues such as pointing jitter, large surface beam diameters, and atmospherics which all combine to raise the measurement noise floor and potentially reduce the measurement accuracy. The Air Force has investigated a wide variety of features, feature selectors and classifiers. Recent work has shown that reasonably accurate measurements of engine RPM and other features can be recovered. It should be noted that unlike acoustic or sonar measurements which utilize far field measurements of a mechanical system a LiDAR vibrometer is actually measuring the surface vibrations of the device itself. The primary challenge is to robustly extract the salient features from the vibrometry signal that remain consistent and meaningful even though the signatures vary between vehicles of the same type. The Air Force will soon be initiating a ground and flight test effort of a prototype tactical range LiDAR system under the Sensor for Image Recognition and Exploitation (SIREN) program. This sensor is capable of making vibration measurements at militarily useful ranges and will be used to collect data for identification algorithm development. These measurements will be made on a variety of instrumented targets to assess the military utility of the vibrometer sensor and to collect measurements for the development of classification and identification algorithms to form the basis of a vibrometer pilot aiding target recognition system.

Objective:

The objective of the CLOVIS effort is to develop, parametrically optimize, and assess extensible algorithms that can classify and identify mechanically operating systems utilizing measurements from an airborne LiDAR vibrometer. The overall goal is to develop a near real-time (i.e. identification result is available <1 second after data is available) suite of robust algorithms that can be efficiently and dynamically programmed to classify and identify a number of mechanical systems for the purpose of demonstrating the combat identification capability of airborne vibration sensors. The parametrically optimized algorithms and their associated parameters must be suitable for executing on the current target processor and operating system dedicated for software demonstrations in the SIREN LiDAR sensor (ORION-7654 Single Board Computer with a Linux operating system). Data for generating the algorithm parameters shall consist of measurements from the SIREN LiDAR system and truth data from accelerometers and tachometers collected from targets of interest. The initial number of targets will be limited to five or six but the algorithms and their associated identification and classification parameters must be extensible to 30 or 40 targets (with solution time <1 second). The algorithms must be able to correctly identify a particular class of vehicle greater than 95% with a confidence level of 90%.

In order to achieve the desired identification performance a multi-disciplinary approach is required. A general understanding of LiDAR systems and how they function at range is necessary in order to understand sensor measurement limitations and at range performance impacts to vibration measurement. An understanding of vehicle acoustic and structural interactions, modal impacts, interactions on surface vibration measurements is also required. Software Systems engineering and software development best practices is required in order to aid in the development and hosting the data and software on the target SBC. This knowledge in necessary in order to develop the necessary robust features to account for a wide variety of operating conditions. Vendor is expected to have an advanced understanding in the areas of signal processing, noise and vibration analysis, acoustics, modal analysis, software development, and system engineering and software system engineering practices.

Since the goal is a software that will successfully classify and identify vehicles from their vibration characteristics the metrics for measuring this performance need to be defined. Assuming the identification or Automatic Target Recognition (ATR) algorithms will be given a query in the form of a velocity or acceleration time-series from one of several predetermined test targets, the performance evaluation will consider several metrics common to pattern recognition tasks. Example metrics are precision and recall/sensitivity assuming a classification decision is made. Such performance values should be presented as precision-recall curves and F-scores. Additionally, estimates of decision confidence could be provided by the ATR to facilitate utilization in a non-forced decision paradigm. Such performance could be illustrated for singular confidence thresholds via confusion matrices with the addition of an "unknown" classification result.

It is desirable that ambiguity in decision, e.g. similar confidences in multiple possible classification results for a single query, be reported following the classification hierarchy (e.g. category, class, type, id). In this fashion, for example, low confidence at a type-level decision might be mitigated by considering the confidence only at the class-level.

The CLOVIS effort will parallel the SIREN program so that multiple spiral deliveries of the software and algorithms are expected as data and knowledge becomes available over the course of the program.

Software source is expected to be in a compilable language such as C++ or Objective C and pseudo code such as that used in Python, Matlab, or Octave. All source is expected as a deliverable as are the parameters used in the algorithms for identification. Government personnel will host the algorithms on the target processer to verify operation before use in testing. An Interface Control Document detailing information available to the algorithms on the SBC will be made available.

The vendor shall also develop a database that organizes and stores the collected test data and associated meta-data to aid in parametrically optimizing and algorithm performance analysis. Data shall include LiDAR measurement, accelerometer, tachometer, and associated meta-data. The database and associated software is also to be delivered to the government.

Scope and Structure:

This is a 48 month effort with an additional 3 months for completion/submission of a final report. The effort will focus on data analysis, algorithm development, and test assessments. Areas of research include statistics, signal analysis, structural analysis, vibration analysis, and software engineering.

Analysis of sensitivity of algorithms to noise and signal variation due to operating conditions is also expected. A non-disclosure between the vendor and sensor vendor is required to support data transfer and to communicate additional data needs for algorithm development to the sensor operator. In addition, the vendor will have to execute a DD-254 to the collateral level. Vendor will be expected to deliver algorithm documentation and pseudo code or actual source for integration into the target single board computer. An Interface Control Document (ICD) and specific hardware will be made available so that vendor understands data storage, processing, and execution rate limitations. Several end products are expected as the end of this effort: The first is a set of flexible, extensible, and sustainable algorithms and associated parameters that can classify and identify mechanically operating targets at a high confidence level. The second is a data collection process that ensure the necessary data is collected to successfully optimize the algorithms to achieve a high level of confidence performance. The third is a data base system that is sustainable, supportable, and extensible to document and store the necessary tuning and test data for the algorithms. The fourth is documentation that makes this process easily transferrable.

Specific Technical and Program Objectives

The vendor shall deliver all software, firmware, data, databases, and associated documentation developed during this effort.

The vendor shall accomplish the following

Basic

1. Accomplish a NDA with the vibrometer sensor prime contractor

2. Accomplish a NDA with the AFRL/RYMM support contractor

3. Accomplish a DD-254 Security form

4. Utilize System and Software Engineering Best Practices

5. Provide program management to administer technical and financial aspects of program

6. Document engineering design and development effort, software, data, and test results.

7. Conduct a Kickoff meeting 30 days after contract award

8. Conduct technical interchanges that document and present interim program status

9. Attend test planning, test execution, and post data analysis events

Technical

1. Conduct review of past activities

2. Develop a software spiral development plan for all algorithms

3. Develop a software spiral development plan for algorithm database develop

4. Execute software and database spiral development plan

5. Coordinate with government on maintenance of Interface Control Documents

6. Develop and maintain software design documents and software user guide documents

7. Develop and execute a Modeling and Simulation plan to assess algorithms

8. Coordinate integration of algorithms with government into target platform

9. Maintain a collateral level processing environment

10. Assess long term requirements for sustainability of database and algorithms

11. Participate in test planning for data collections

12. Participate in test events and conduct data analysis for quality and algorithm development

13. Write test reports to assess and document data collects

14. Develop a software test plans, conduct software tests, and test reports

15. Support development of flight test plans focused on algorithm data collects

16. Write test reports assessing algorithm performance during flight tests

17. Coordinate with government and partners on software development and test events

18. Coordinate and support government on integration of algorithms/software into target SBC

Management Objective:

Vendor shall be permitted flexibility to manage cost, schedule, and performance of the contract.

Non-Data Deliverable (Outputs):

Hardware: No hardware development planned for this effort

Software Database: Software database and associated libraries developed and modified during the course of this effort shall be delivered to the government.

Software Code: All software pseudo code, interpreter code, source, and associated libraries will be delivered to the government.

Security Objective:

General Operations Security (OPSEC) procedures, policies, and awareness are required in an effort to reduce program vulnerability from unfriendly collection and exploitation of critical information. OPSEC will be applied throughout the life cycle of the contract. While working on a government installation, OPSEC guidance will be provided by the RYOY Information Protection Office. In addition, a DD-254 will be accomplished and the vendor is required to implement and maintain a collateral level storage and processing capability. The contractor shall adhere to the Electro-Optical Targeting Sensors Security Classification/Declassification Guidance dated 15 April 1990 Revised 20 February 2015.

Safety Objective:

The following are the Government’s safety objective for this effort:

1. General office and ergonomic safety and associated training will be followed and administered.

2. The vendor shall enact safety policies and procedures which will mitigate and risk of injury/illness to employees and visiting government personnel.

3. To the extent required by the contract or grant, the vendor or university shall implement a safety plan to mitigate the office and computer laboratory area safety hazards.

Schedule Objectives

The contract is a 48 month effort, plus 3 months for completion/submission of a final report.

Cost Objective:

Funding Source: PE 63742 Combat Identification

FY17 FY18 FY19 FY20 FY21 Total:

700K 1,500K 1000K 900K 800K 4,900K