Explainable GeoAI: can saliency maps help interpret artificial intelligence’s learning process? An empirical study on natural feature detection

AbstractImproving the interpretability of geospatial artificial intelligence (GeoAI) models has become critically important to open the 'black box' of complex AI models, such as deep learning. This paper compares popular saliency map generation techniques and their strengths and weaknesses in interpreting GeoAI and deep learning models' reasoning behaviors, particularly when applied to geospatial analysis and image processing tasks. We surveyed two broad classes of model explanation methods: perturbation-based and gradient-based methods. The former identifies important image areas, which help machines make predictions by modifying a localized area of the input image. The latter evaluates the contribution of every single pixel of the input image to the model's prediction results through gradient backpropagation. In this study, three algorithms—the occlusion method, the integrated gradients method, and the class activation map method—are examined for a natural feature detection task using deep learning. The algorithms' strengths and weaknesses are discussed, and the consistency between model-learned and human-understandable concepts for object recognition is also compared. The experiments used two GeoAI-ready datasets to demonstrate the generalizability of the research findings.Keywords: XAIartificial intelligencedeep learningvisualizationGeoAI Disclosure statementNo potential conflict of interest was reported by the author(s).Data and codes availability statementThe data and codes that support the findings of this study are available at https://github.com/ASUcicilab/explainable-geoai. Instructions on how to use the data and codes are provided in the README file.Additional informationFundingThis work is supported in part by the National Science Foundation under [awards 2120943, 2230034, 1853864].Notes on contributorsChia-Yu HsuChia-Yu Hsu is a research professional at Arizona State University. His research interests include artificial intelligence, computer vision, spatiotemporal data analysis, and their applications in climate change and terrain research.Wenwen LiWenwen Li is a professor in geographic information science at Arizona State University (ASU). Her research interests are cyberinfrastructure, big data, GeoAI and their applications in data- and computation-intensive environmental and social sciences. At ASU, she directs the Cyberinfrastructure and Computational Intelligence Lab (http://cici.lab.asu.edu/) and serves as the Research Director for the Spatial Analysis Research Center.