Multimodal Intelligence: Representation Learning, Information Fusion, and Applications

Deep learning methods have revolutionized speech recognition, image\nrecognition, and natural language processing since 2010. Each of these tasks\ninvolves a single modality in their input signals. However, many applications\nin the artificial intelligence field involve multiple modalities. Therefore, it\nis of broad interest to study the more difficult and complex problem of\nmodeling and learning across multiple modalities. In this paper, we provide a\ntechnical review of available models and learning methods for multimodal\nintelligence. The main focus of this review is the combination of vision and\nnatural language modalities, which has become an important topic in both the\ncomputer vision and natural language processing research communities. This\nreview provides a comprehensive analysis of recent works on multimodal deep\nlearning from three perspectives: learning multimodal representations, fusing\nmultimodal signals at various levels, and multimodal applications. Regarding\nmultimodal representation learning, we review the key concepts of embedding,\nwhich unify multimodal signals into a single vector space and thereby enable\ncross-modality signal processing. We also review the properties of many types\nof embeddings that are constructed and learned for general downstream tasks.\nRegarding multimodal fusion, this review focuses on special architectures for\nthe integration of representations of unimodal signals for a particular task.\nRegarding applications, selected areas of a broad interest in the current\nliterature are covered, including image-to-text caption generation,\ntext-to-image generation, and visual question answering. We believe that this\nreview will facilitate future studies in the emerging field of multimodal\nintelligence for related communities.\n