Reciprocal Human-Machine Learning: A Theory and an Instantiation for the Case of Message Classification

There is growing agreement among researchers and developers that in certain machine-learning (ML) tasks, it may be advantageous to keep a “human in the loop” rather than rely on fully autonomous systems. Continual human involvement can mitigate machine bias and performance deterioration while enabling humans to continue learning from insights derived by ML. Yet a microlevel theory that effectively facilitates joint and continual learning in both humans and machines is still lacking. To address this need, we adopt a design science approach and build on theories of human reciprocal learning to develop an abstract configuration for reciprocal human-ML (RHML) in the context of text message classification. This configuration supports learning cycles between humans and machines who repeatedly exchange feedback regarding a classification task and adjust their knowledge representations accordingly. Our configuration is instantiated in Fusion, a novel technology artifact. Fusion is developed iteratively in two case studies of cybersecurity forums (drug trafficking and hacker attacks), in which domain experts and ML models jointly learn to classify textual messages. In the final stage, we conducted two experiments of the RHML configuration to gauge both human and machine learning processes over eight learning cycles. Generalizing our insights, we provide formal design principles for the development of systems to support RHML. This paper was accepted by D. J. Wu, Special Issue on the Human-Algorithm Connection. Funding: This work was supported by the Israel’s Ministry of Defence [Grant R4441197567] and the Israel’s Ministry of Science and Technology [Grant 207076]. Supplemental Material: The data files are available at https://doi.org/10.1287/mnsc.2022.03518 .