Do Explainable AI techniques effectively explain their rationale? A case study from the domain expert’s perspective

Artificial Intelligence (AI) systems are technologies impacting our lives. The systems learn from existing datasets that record past human decisions. Their performance is measured in terms of accuracy, precision, and recall for reproducing already-known results. Understanding the system's rationale is crucial to check for bias and accept such technology. Explainable AI (XAI) is the area devoted to opening the AI black box, and designing guidelines to build explainable AI systems. Nevertheless, it is important to understand the user's needs for these explanations. This paper presents an investigation of the usefulness of XAI systems in the field of cancer diagnosis from the domain expert's (oncologist) perspective. The main findings suggest domain experts (1) understood the outcomes of the XAI systems; (2) considered XAI outcomes as informative, rather than explanatory; (3) would like to go beyond the fixed presented perspective; and (4) missed the causal relation that would reveal the system's rationale.