Executive-centered AI? Designing predictive systems for the public sector

Recent policies and research articles call for turning AI into a form of IA ('intelligence augmentation'), by envisioning systems that center on and enhance humans. Based on a field study at an AI company, this article studies how AI is performed as developers enact two predictive systems along with stakeholders in public sector accounting and public sector healthcare. Inspired by STS theories about values in design, we analyze our empirical data focusing especially on how objectives, structured performances, and divisions of labor are built into the two systems and at whose expense. Our findings reveal that the development of the two AI systems is informed by politically motivated managerial interests in cost-efficiency. This results in AI systems that are (1) designed as managerial tools meant to enable efficiency improvements and cost reductions, and (2) enforced on professionals on the 'shop floor' in a top-down manner. Based on our findings and a discussion drawing on literature on the original visions of human-centered systems design from the 1960s, we argue that turning AI into IA seems dubious, and ask what human-centered AI really means and whether it remains an ideal not easily realizable in practice. More work should be done to rethink human-machine relationships in the age of big data and AI, in this way making the call for ethical and responsible AI more genuine and trustworthy.