A unified deep learning framework for urban functional zone extraction based on multi-source heterogeneous data

Remote sensing imagery (RSI) and point of interest (POI) are two complementary data for urban functional zone (UFZ) extraction. However, current methods only use single data or just simply fuse the features extracted from these two data, which may not fully exploit their complementary strength. To solve this problem, in this paper, we propose a unified deep learning framework containing two modules to jointly use RSI and POIs for UFZ extraction. In the first module, the complementary feature learning and fusing module, two coupled convolutional neural networks (CNNs) are used to learn the visual feature from RSI and social feature from POIs, respectively. Specifically, to apply CNN on discrete POIs, we convert them into a hierarchical distance heatmap, and employ CNN with the attention mechanism to extract the co-occurrence relationship of POIs in a specific UFZ for social feature representation. Afterward, we ensemble these two coupled CNNs by a feature attention based fusion strategy, to fuse the complementary features learned from RSI and POIs with adaptively learned weights in an end-to-end manner. In the second module, the UFZ spatial relationship modeling module, different from previous methods that only consider features in single UFZ, we design a spatial relation learning network, which can aggregate both local and long-range between-UFZ spatial relationship for UFZ classification. Experiments on three urban regions demonstrate that the proposed framework can take full advantage of visual, social, and spatial features learned from both RSI and POIs, and thus achieve more satisfactory result than the methods using single data or simple fusion strategy. Furthermore, we analyze the impacts of several factors on UFZ extraction, including the contributions of different category of POIs on UFZ extraction improvement, the synergy mechanism of RSI and POIs for the classification of UFZs, and the influence of using different mapping unit for UFZ mapping. The insights distilled from this study can potentially help researchers use heterogeneous data for UFZ extraction. The source codes are available at: https://github.com/GeoX-Lab/UnifiedDL-UFZ-extraction