Image description through fusion based recurrent multi-modal learning

Current research in computer vision and machine learning has demonstrated some great abilities at detecting and recognizing objects in natural images. The promising results in these areas have inspired research towards solving more complex multi-modal learning problems in the image/video domains such as automatic annotation, segmentation, labelling, and generic understanding. Although solutions have been provided for one or more of these problems, their approaches have been application specific. This paper introduces an end-to-end trainable Fusion-based Recurrent Multi-Modal (FRMM) model to address multi-modal applications. FRMM allows each input modality to be independent in terms of architecture, parameters and length of input sequences. FRMM image description models seamlessly blend convolutional neural network feature descriptors with sequential language data in a recurrent framework. For training and testing we used the Flickr30K and MSCOCO datasets, demonstrating state-of-the-art description results.