Optical generative models

Abstract Generative models cover various application areas, including image and video synthesis, natural language processing and molecular design, among many others 1–11 . As digital generative models become larger, scalable inference in a fast and energy-efficient manner becomes a challenge 12–14 . Here we present optical generative models inspired by diffusion models 4 , where a shallow and fast digital encoder first maps random noise into phase patterns that serve as optical generative seeds for a desired data distribution; a jointly trained free-space-based reconfigurable decoder all-optically processes these generative seeds to create images never seen before following the target data distribution. Except for the illumination power and the random seed generation through a shallow encoder, these optical generative models do not consume computing power during the synthesis of the images. We report the optical generation of monochrome and multicolour images of handwritten digits, fashion products, butterflies, human faces and artworks, following the data distributions of MNIST 15 , Fashion-MNIST 16 , Butterflies-100 17 , Celeb-A datasets 18 , and Van Gogh’s paintings and drawings 19 , respectively, achieving an overall performance comparable to digital neural-network-based generative models. To experimentally demonstrate optical generative models, we used visible light to generate images of handwritten digits and fashion products. In addition, we generated Van Gogh-style artworks using both monochrome and multiwavelength illumination. These optical generative models might pave the way for energy-efficient and scalable inference tasks, further exploiting the potentials of optics and photonics for artificial-intelligence-generated content.