'Computer Says No!': The Impact of Automation on the Discretionary Power of Public Officers

The goal of this paper is to unpack the “human in the loop” requirement in the process of automation. It will analyze the impact of automation on street-level bureaucrats, and lay out the steps policy makers need to take into account to¸ ensure that meaningful human discretion is maintained. This issue will be examined comparatively through two cases related to the use of automation to detect and investigate fraud in welfare benefits, that have been brought recently before the court. The first algorithm is deployed in Michigan by the Unemployment Insurance Agency for detecting and investigating unemployment fraud. This algorithm is a decision making algorithm with very broad and draconian capabilities including deciding on cutting individuals’ benefits. The second algorithm is deployed in the Netherlands by the Ministry of Social Affairs and Employment to detect different types of welfare fraud. This algorithm is a decision aiding algorithm that automates the process of data collection and analysis, but the final determination is kept in the hands of the relevant fraud examiner. The paper concludes that both algorithms could do more to maintain meaningful discretion. In Michigan, after the automation, the room left for street-level bureaucrats to apply discretion is very narrow. Thus, it is suggested that the automated tasks should be narrowed down to include only few segments of the unemployment fraud detection and investigation process. In contrast, the Netherlands’ algorithm maintains more room for discretion for street-level bureaucrats to weigh in. The discretion is also more meaningful, as it is clearer what the role of the human in the loop is. However, since the algorithm is de-facto deciding who will be investigated, more cautionary steps ensuring that humans are involved at the right part of the process should be taken. In addition, the paper points out that lack of human agency was only one reason for the failures of the algorithms. There are two additional causes, the first related to poor technical design, and the second related to the socio-political context that the algorithms operate in. This classification that will be discussed in the paper is important for understanding which failures are associated with which reasoning. Although it became apparent that keeping human in the loop in the automation process is important, the cases demonstrate that questions such as: what should be the role of the human, and how to design the human algorithm interaction, have not received sufficient attention in the literature. Therefore, the paper aims to shed light on these issues.