Deep color calibration for UAV imagery in crop monitoring using semantic style transfer with local to global attention

• A multi decoder network that integrates semantic segmentation and style transfer. • An AdaIN method to estimate the color cast in the crop areas. • Local to global attention to estimate the color cast outside the crop areas. Color cast of the UAV imagery is inevitable due to the temperature and illuminance changes during the UAV flight. Since color is an important feature in crop monitoring, color cast often produces misleading inference in estimation of crop stress, nutrient, and productivity. Therefore, color calibration is a necessary step to remove the negative effects of color variation for UAV based crop monitoring. Nowadays, the state of art color calibration methods usually use the semantic correspondences for accurate color transfer. However, the mainstream color calibration methods ignore the integration of semantic segmentation and style transfer, and suffer the problem of semantic mismatch. To address this problem, this study proposed a multi decoder architecture that builds the integration of sematic segmentation and style transfer for the color transfer in an end to end mode. Also, this paper introduced an Crop Oriented adaptive instance normalization (AdaIN) method to estimate the color cast in the crop areas, and used that estimated information for color calibration over the whole image area with a local to global attention mechanism. Each proposed module was evaluated in ablation study to test its effectiveness, respectively. Also, the proposed method was evaluated on several crop types and compared with the state of art methods. Experimental results showed that our proposed method achieved state of art or close to state of art performance in all metrics. The research of this work is expected to obtain a general framework to remove the color cast of UAV imagery for crop monitoring, which may build a good foundation for the following data interpretation. Our dataset and codes are can be downloaded at http://github.com/huanghsheng/deep-color-calibration .