Human-Algorithm Collaborative Truth Inference in Crowdsourcing

众包 推论 机器学习 计算机科学 基本事实 超参数 稳健性(进化) 水准点(测量) 归纳偏置 公制(单位) 骨料(复合) 互补性(分子生物学) 人类智力 人工智能 数据挖掘 聚合问题 众包软件开发 合成数据 性能指标 训练集
作者
Xuan Wei,Mingyue Zhang,Qingpeng Zhang,Zhi Li,Daniel Zeng
出处
期刊:Informs Journal on Computing [Institute for Operations Research and the Management Sciences]
标识
DOI:10.1287/ijoc.2023.0440
摘要

Crowdsourcing has become a pivotal strategy in gathering large-scale, high-quality labeled data, particularly in data-intensive applications powered by artificial intelligence. To aggregate the noisy crowd efforts, many studies have considered learning a predictive algorithm based on the noisy human annotations and subsequently integrating the learned knowledge back into the data aggregation process. However, it is unclear how to design such hybrid systems that maximize the complementary strengths of humans and algorithms. In response, we analyze the patterns of human and algorithm intelligence and propose that the inductive bias of algorithms can effectively mitigate inconsistencies in human labeling, thus complementing human efforts. Building on this premise, we propose a human-algorithm collaborative framework (HAC) to combine human labels with algorithmic predictions. By proposing a metric called hybrid complementarity score (HCS) to quantify human-algorithm complementarity, our framework can dynamically adjust the weight of each algorithm based on its complementarity, significantly enhancing the overall efficacy of the human-algorithm integration. To validate the effectiveness of our framework, we first instantiate it with several algorithms, including a high-complementarity algorithm building upon the inductive bias of clustering-aware design. We then benchmark our framework against leading baselines across eight real-world tasks. Our results not only demonstrate the superior performance of our proposed framework but also affirm its robustness across different algorithm selections (e.g., types and number of algorithms) and hyperparameter configurations. This research not only delivers a feasible and effective solution for truth inference in crowdsourcing but also contributes to the burgeoning community of human-algorithm collaboration. History: Accepted by Ram Ramesh, Area Editor for Data Science and Machine Learning. Funding: X. Wei is supported by the National Natural Science Foundation of China (NSFC) [Grants 72201167, 72192822, 72571175, 72331006, 72221001, and 72232005] and the Young Elite Scientists Sponsorship Program by CAST [Grant 2023QNRC001]. D. D. Zeng is supported by NSFC [Grant 72293575]. M. Zhang is supported by NSFC [Grant 72272101]. Q. Zhang is supported by the General Research Fund of the Research Grant Council of Hong Kong [Grant 17209225]. X. Wei also thanks the Science and Technology Commission of Shanghai Municipality [Grant 22JC1403600]. Supplemental Material: The software that supports the findings of this study is available within the paper and its Supplemental Information ( https://pubsonline.informs.org/doi/suppl/10.1287/ijoc.2023.0440 ) as well as from the IJOC GitHub software repository ( https://github.com/INFORMSJoC/2023.0440 ). The complete IJOC Software and Data Repository is available at https://informsjoc.github.io/ .
最长约 10秒,即可获得该文献文件

科研通智能强力驱动
Strongly Powered by AbleSci AI
科研通是完全免费的文献互助平台,具备全网最快的应助速度,最高的求助完成率。 对每一个文献求助,科研通都将尽心尽力,给求助人一个满意的交代。
实时播报
刚刚
sparks完成签到,获得积分10
刚刚
shuyiWang发布了新的文献求助10
1秒前
牧青发布了新的文献求助10
1秒前
Orange的应助被念汐采纳,获得10
1秒前
事成发布了新的文献求助10
1秒前
mlx完成签到 ,获得积分10
1秒前
sunziyi发布了新的文献求助30
2秒前
许琦发布了新的文献求助10
3秒前
3秒前
小蘑菇的应助被发了十篇Nature采纳,获得10
3秒前
CipherSage的应助被Firsterchao采纳,获得10
3秒前
3秒前
666y完成签到,获得积分10
4秒前
怕黑的乌完成签到,获得积分10
5秒前
聪明紫山完成签到,获得积分10
5秒前
JamesPei的应助被刘金磊采纳,获得10
5秒前
火山发布了新的文献求助10
5秒前
ubless607的应助被medsci采纳,获得10
5秒前
Az的应助被medsci采纳,获得10
6秒前
科研通AI6.3的应助被medsci采纳,获得10
6秒前
贾贡献的应助被medsci采纳,获得10
6秒前
滴滴发布了新的文献求助10
6秒前
共享精神的应助被medsci采纳,获得10
6秒前
lx的应助被medsci采纳,获得20
6秒前
在水一方的应助被medsci采纳,获得10
6秒前
香蕉觅云的应助被medsci采纳,获得10
7秒前
7秒前
rodion完成签到 ,获得积分10
7秒前
Orange的应助被medsci采纳,获得10
7秒前
无极微光的应助被medsci采纳,获得20
7秒前
Owen的应助被pivot_literature采纳,获得10
7秒前
TOMORROW完成签到,获得积分10
8秒前
8秒前
见忧的应助被yan采纳,获得10
8秒前
9秒前
忧郁的怜珊完成签到,获得积分20
9秒前
森林木完成签到,获得积分10
10秒前
隐形曼青的应助被豆豆采纳,获得10
10秒前
前撅狼完成签到,获得积分10
10秒前
高分求助中
(应助此贴封号)【重要!!请各用户(尤其是新用户)详细阅读】【科研通的精品贴汇总】 10000
拉伸高分子熔体中缠结行为的模拟和小角中子散射研究 2000
Agyptische Geschichte der 21.30. Dynastie 510
高分子基功能界面水蒸发材料与应用研究 500
外场增强的水汽发电研究 500
基于结构调控的功能化水诱导发电材料与器件研究 500
基于界面调控的高效湿气发电材料与器件研究 500
热门求助领域 (近24小时)
化学 材料科学 医学 生物 纳米技术 计算机科学 工程类 有机化学 化学工程 生物化学 物理 复合材料 内科学 催化作用 光电子学 人工智能 细胞生物学 无机化学 物理化学 数学
热门帖子
关注 科研通微信公众号,转发送积分 8191812
求助须知:如何正确求助?哪些是违规求助? 9578503
关于积分的说明 21859856
捐赠科研通 7791184
什么是DOI,文献DOI怎么找? 3436733
关于科研通互助平台的介绍 2560447
邀请新用户注册赠送积分活动 2483502