Lite2Relight: 3D-aware Single Image Portrait Relighting

Achieving photorealistic 3D view synthesis and relighting of human portraits\nis pivotal for advancing AR/VR applications. Existing methodologies in portrait\nrelighting demonstrate substantial limitations in terms of generalization and\n3D consistency, coupled with inaccuracies in physically realistic lighting and\nidentity preservation. Furthermore, personalization from a single view is\ndifficult to achieve and often requires multiview images during the testing\nphase or involves slow optimization processes.\n This paper introduces Lite2Relight, a novel technique that can predict 3D\nconsistent head poses of portraits while performing physically plausible light\nediting at interactive speed. Our method uniquely extends the generative\ncapabilities and efficient volumetric representation of EG3D, leveraging a\nlightstage dataset to implicitly disentangle face reflectance and perform\nrelighting under target HDRI environment maps. By utilizing a pre-trained\ngeometry-aware encoder and a feature alignment module, we map input images into\na relightable 3D space, enhancing them with a strong face geometry and\nreflectance prior.\n Through extensive quantitative and qualitative evaluations, we show that our\nmethod outperforms the state-of-the-art methods in terms of efficacy,\nphotorealism, and practical application. This includes producing 3D-consistent\nresults of the full head, including hair, eyes, and expressions. Lite2Relight\npaves the way for large-scale adoption of photorealistic portrait editing in\nvarious domains, offering a robust, interactive solution to a previously\nconstrained problem. Project page:\nhttps://vcai.mpi-inf.mpg.de/projects/Lite2Relight/\n