4F optical neural network acceleration: an architecture perspective

Low latency, high throughput inference on Convolution Neural Networks (CNNs) remains a challenge, especially for applications requiring large input or large kernel sizes. 4F optics provides a solution to potentially accelerate CNN inferences with Fourier optics and the well-known convolution theorem. However, existing 4F CNN accelerators suffer from various limitations that make the implementation of a multi-channel, multi-layer CNN not scalable or even impractical. In this paper, we discuss the limitations of 4F CNN accelerators including the positive sensor readout, intensity-only modulation and slow modulation frequency and methods to address them. We also propose the channel tiling method that can address an important throughput and precision bottleneck of high-speed, massively-parallel optical 4F computing systems, not requiring any additional optical hardware.