Visual Sensing from MultiModal Spike-based Learning using Tactile Sensors

—This work contributes an event-driven visual-tactileperception system, comprising a novel biologically-inspired tactilesensor and multi-modal spike-based learning. Our neuromorphicfingertip tactile sensor, NeuTouch, scales well with the number oftaxels thanks to its event-based nature. Likewise, our VisualTactile Spiking Neural Network (VT-SNN) enables fast perceptionwhen coupled with event sensors. We evaluate our visual-tactilesystem (using the NeuTouch and Prophesee event camera) on tworobot tasks: container classification and rotational slip detection.On both tasks, we observe good accuracies relative to standarddeep learning methods. We have made our visual-tactile datasetsfreely-available to encourage research on multi-modal eventdriven robot perception, which we believe is a promising approachtowards intelligent power-efficient robot systems.