Object Hardness Estimation With Optical Tactile Sensor Using Spiking Neural Network

Hardness is a crucial physical property in robotic perception and manipulation. Optical tactile sensors offer an effective approach for hardness perception by capturing high-resolution images of contact-induced deformation. Inspired by biological information processing mechanisms, we propose a Hardness Estimation Spiking Neural Network (HE-SNN) model for simultaneous objective hardness regression and subjective hardness classification. The model converts dynamic tactile image sequences, containing both RGB and optical flow data from a GelSight Mini sensor, into spatiotemporal spike trains and processes them via a dual-stream SNN architecture. Validation on diverse datasets (silicone, textured composites, natural objects) demonstrates high predictive accuracy and robust generalization, comparable to state-of-the-art methods. Furthermore, the model is successfully applied to a fully automated robotic kiwifruit sorting task, showcasing its practical utility in a real-world scenario. This study confirms the SNN approach’s effectiveness for multi-task tactile perception, highlighting its potential for future robotic systems.