Reflections on Automation Complacency

AbstractAutomation is replacing or working alongside human operators in various settings. However, its widespread implementation poses potential risks for operators, one of which is referred to as "automation complacency" or simply "complacency." This concept originated in the field of aviation crash investigations and has since been developed and researched in human-automation interaction (HAI). It holds professional operators in safety-critical systems accountable for system failures. The increasing prevalence of vehicle automation has shed more light on this issue, as it has been identified as a probable cause of recent traffic crashes involving partial and conditional automation. Furthermore, it even plays a significant role in judicial decision-making. In this position paper, we delve into seven key questions surrounding this concept, including its conceptualization, operationalization, detectability beforehand, and prevention. We also examine its causal influence on traffic crashes and its potential to be perceived as nothing more than a blame game. When subjecting these questions to critical scrutiny, we argue that while it is an important phenomenon in HAI, there are valid concerns regarding its appropriate usage in crash investigations and liability litigation. Applying it uncritically in these fields may give rise to moral and ethical issues and result in new consumer harm.Keywords: Human-automation interactionvehicle automationautomation complacencyhindsight biasblame game AcknowledgementsI thank Prof. Zhizhong Li and Yueying Chu for providing helpful feedback on earlier versions of this article. The views and opinions expressed in this paper are those of the author.Disclosure statementNo potential conflict of interest was reported by the author(s).Additional informationFundingThis work is supported by the National Natural Science Foundation of China [Grant No. 72071143 and T2192932].Notes on contributorsPeng LiuPeng Liu is with the Center for Psychological Sciences, Zhejiang University, China. His research is focused on human factors and ergonomics, risk analysis, and risk perception related to complex systems such as automated driving and nuclear power plants.