AI advice and human metacognition

Although organizations increasingly rely on artificial intelligence (AI) to support decision making, research consistently finds that decision makers exhibit varied responses to algorithmic advice. While prior research has examined when users are more likely to accept or reject AI advice, little is known about the cognitive factors associated with these responses or the broader psychological consequences of working with AI. Drawing on metacognition theory, this study explores how individuals monitor and revise their judgments when confronted with algorithmic advice in a behavioral experiment with 440 participants. The results indicate that metacognitive estimates of confidence are strongly associated with responses to algorithmic advice. However, individual differences in metacognitive sensitivity shape how consistently they rely on confidence when responding to AI advice. Specifically, when metacognitive sensitivity is low, confidence judgments are based on noisy internal states, which, in turn, lead to inconsistent patterns of advice-taking. We also find that exposure to contradictory AI advice is associated with lower retrospective confidence in individuals' subsequent decisions. This loss of confidence is more pronounced when individuals reject the AI advice, highlighting a hidden psychological cost of advice rejection. These findings contribute to the growing body of information systems (IS) research on algorithmic advice-taking by demonstrating that effective human-AI collaboration requires not only improvements in system design but also fostering users' metacognitive sensitivity.