CPS-CIR, CPS-FR: hiddenobjects: physics-guided latent space models for detecting occluded objects

The National Science Foundation Division of Computer and Network Systems awarded South Dakota State University one million dollars on September 1, 2026, under the Computer and Information Science and Engineering program (CFDA 47.070) to develop physics-guided latent space models for detecting occluded objects in agricultural settings.

The project, titled "Hidden Objects: Physics-Guided Latent Space Models for Detecting Occluded Objects," addresses the challenge of robotic and camera-based systems failing to detect partially hidden objects such as ripe fruit obscured by leaves and branches. The research team will develop a thermal sensing approach that applies controlled heat pulses to plant regions and uses thermal cameras to observe cooling patterns over time. Because fruit, leaves, stems, and surrounding environments absorb and release heat differently, their unique thermal signatures provide information unavailable from standard imagery. The project integrates multimodal sensing hardware—synchronized thermal, color, depth, and near-infrared data—with physics-guided machine learning algorithms and a robotic test platform. By analyzing temporal thermal evolution following heat pulses and applying physical principles of heat diffusion and cooling to guide machine learning, the approach aims to improve detection robustness to occlusion, lighting changes, and variable outdoor conditions.

The award runs from September 1, 2026, through August 31, 2030. Performance occurs in Brookings, South Dakota. The assistance type is a Project Grant.

Abstract

Many robots and camera-based systems struggle to detect objects when they are partially hidden from view. This challenge is important in agriculture, where leaves and branches can block the view of ripe fruit and cause conventional cameras to miss valuable information. Undetected fruit can reduce harvesting efficiency, increase labor demands, and limit the effectiveness of robotic harvesting technologies. This CPS project develops an approach for robots to identify hidden fruit by using heat as an additional source of information. The proposed system applies a short and controlled heat pulse to a small region of a plant. IT then uses a thermal camera to observe how the region cools over time. Fruit, leaves, stems, and the surrounding environment absorb and release heat differently. Their unique thermal responses can provide information that is not available from standard images. By analyzing these cooling patterns using machine learning, robots can identify objects that are partially or completely hidden from direct view. This project advances scientific understanding of how thermal information, physics-based models, and machine learning methods for agricultural cyber-physical systems. This research develops physics-guided computer vision methods to detect occluded objects that uses synchronized thermal, color, depth, and near-infrared data. The central idea is to analyze not only an object's temperature but also its temporal evolution following a heat pulse or a natural heating event. The models use physical principles of heat diffusion and cooling to guide machine learning and improve robustness to occlusion, changing lighting, and variable outdoor conditions. The project develops multimodal sensing hardware, physics-guided learning algorithms, along with a robotic test platform that can select regions of interest, and collect thermal video. Using the detections, IT supports robot grasping decisions. Experiments begin with simulations and laboratory testbeds, then proceeds to field data collection and trials at partner sites. The project also evaluates crop safety by testing whether small thermal pulses affect ripening, firmness, or post-harvest quality. A dataset is developed containing synchronized thermal, color, depth, and near-infrared sequences from simulations, laboratory experiments, and multi-site field environments. The dataset supports reproducible evaluation of methods for hidden-object detection and physics-guided robotic perception. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the foundation's intellectual merit and broader impacts review criteria.- subawards are planned for this award.