How much AI is too much in assisting evaluation? Chinese applicant reactions to conjoint decisions by experts and algorithm

Purpose As the use of AI techniques becomes more prevalent in evaluation activities, researchers have suggested integrating AI and human evaluators to address potential AI aversions from applicants. This study was conducted in response to this call, utilizing a Chinese sample. Design/methodology/approach This study examined applicants’ responses to decisions made by a human–AI hybrid jury with varying levels of AI agency. Using a three (AI agency: low, medium and high) × 2 (outcome favorability: favorable and unfavorable) between-subjects design, we analyzed the moderated mediation relationship between machine heuristics and organizational assessment, with fairness perceptions serving as a mediator and AI agency level and outcome favorability as moderating variables. Findings The results revealed that positive machine heuristics enhanced fairness perception only at high AI agency levels, irrespective of outcome favorability. Negative machine heuristics impaired fairness perception at medium or high AI agency levels only in cases where participants failed the test. Machine heuristics did not directly influence participants’ assessment of the organization but could indirectly influence organization assessments through fairness perception, contingent on the composition of AI agency level and outcome favorability. Originality/value By having participants go through a real intelligence exam and explore their experiences, this study had theoretical and practical implications for AI evaluation.