Omnidirectional Visual Computing: Foundations, Challenges, and Applications

Omnidirectional images and videos are widespread due to the popularization of devices for capturing and visualization. Unlike pinhole-based imagery, omnidirectional media lie on the surface of a sphere, have a full field of view (FoV), and store the light intensities from an entire environment. Notably, applications involving immersive augmented, mixed, and virtual reality experiences benefit from the full FoV. Although defined on the spherical domain, omnidirectional images are often mapped to a (multi-)planar representation, which results in distorted images and degrades the performance of most traditional visual computing algorithms designed to work on the plane. This tutorial paper reviews the spherical camera model, the most common capturing devices, and popular (multi-)planar representations of omnidirectional media. It also approaches the main challenges of omnidirectional visual computing, focusing on the deep learning paradigm, and tackles four visual computing applications that strongly explore the potential of omnidirectional imagery.