NEOSTI - a neuromorphic electronic-opto spatial-temporal hybrid image sensor

Image sensors in machine vision systems face significant challenges related to energy efficiency and processing capability when storing, transferring, and processing massive amounts of data. In humans, over 80% of brain-processed information is obtained through the eyes, which are capable of detecting and synchronously processing information with extremely low overall power consumption. Inspired by the biomimetics, we propose a Neuromorphic Electronic-Opto Spatial Temporal Imager (NEOSTI), one of the smallest electronic-opto fully integrated, eye-sized vision systems enabling acquisition and operation in typical indoor/outdoor non-coherent environments, under both natural and artificial lighting conditions without any extra requirement of the light source. NEOSTI combines processing-pre-sensor in optical domain, processing-in-sensor with nonlinear acquisition capability while optical to electronic converting, and processing-near-sensor in electronic domain, enabling parallel data computing capabilities while sensing. NEOSTI also integrates a low complexity Binary Neural Network to process image semantic information. It attains competitive performance in several visual processing tasks. The authors demonstrate a neuromorphic imaging system that combines optical and electronic, spatial and temporal processing near the sensor, enabling parallel sensing and computation in typical indoor and outdoor environments without any light source requirements.