Bringing ViTs to the edge: photonic in-sensor compression and parallel computation for ultra-efficient inference

Running Vision Transformers (ViTs) on edge devices is difficult due to the high bandwidth, energy, and latency required to deliver large weight sets to the sensor, as well as the limited on-chip memory available for storing these parameters. To address these limitations, in this position paper, we review one of our promising software–hardware co-design framework that integrates an in-sensor compressed weight retrieval and computation technique into an intelligent vision sensor on Silicon Photonics (SiPh) substrate. The framework employs a hardware-friendly weight compression algorithm that substantially reduces memory footprint, bandwidth demand, and overall power consumption. Besides, it utilizes the efficiency of SiPh devices to enable in-sensor retrieval of compressed weights and parallel, fine-grained convolution operations directly beside the pixel array. This co-designed architecture enables low-power, adaptable ViT inference tailored for constrained edge platforms.