The labor of search engine evaluation: Making algorithms more human or humans more algorithmic?

While search engines are discursively framed as automated, self-governing machines, they remain dependent on human laborers. This behind-the-screen workforce includes not only programmers and engineers but also categories of workers that have drawn far less attention. The latter include search quality raters, contingent workers tasked with supplying a “human check” on search algorithms. To explore the human–machine entanglements in search engine evaluation, we draw upon 21 in-depth interviews with raters located across the globe; we supplement our interview data with an analysis of worker training documents. Our findings reveal that despite Google’s stated efforts to make algorithmic systems more sensitive to human subtlety, raters experience their work in ways that are—paradoxically—algorithmic. Indeed, workers framed their experiences through regimes of standardization, discipline, and invisibility. We conclude by discussing the broader implications of our analysis, including worker dehumanization and the exacerbation of power hierarchies in the global, on-demand economy.