Truth Inference in Crowdsourcing: Is the Problem Solved?

Summary: Comprehensive survey and benchmark of 17 crowdsourcing truth-inference algorithms on five real datasets, with public code/data. Finds strong dataset dependence and no consistently superior method, showing truth inference remains unsolved and identifying research directions. (summarized by gpt-5.6-luna on Jul 24 2026)

Paper ID: h35033f515c359b11
Venue: VLDB
Year: 2017
Pagerank: 7.5691557e-05
Overall Rank: 3,171 | 78.69%
DOI: 10.14778/3055540.3055547

Incoming Non-self Citations Over Time

Authors

1. Yudian Zheng (University of Hong Kong)
2. Guoliang Li (Tsinghua University)
3. Yuanbing Li (Tsinghua University)
4. Caihua Shan (University of Hong Kong)
5. Reynold Cheng (University of Hong Kong)

BibTeX Citation

@article{zheng_vldb17,
        title = {{Truth Inference in Crowdsourcing: Is the Problem Solved?}},
        author = {Zheng, Yudian and Li, Guoliang and Li, Yuanbing and Shan, Caihua and Cheng, Reynold},
        journal = {PVLDB},
        series = {{VLDB} '17},
        volume = {10},
        number = {5},
        pages = {541--552},
        doi = {10.14778/3055540.3055547},
        url = {https://doi.org/10.14778/3055540.3055547},
        year = {2017}
}

Incoming Citations (Sorted by Pagerank)

Showing 18 of 18 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
2,908	AI Meets Database: AI4DB and DB4AI	2021	SIGMOD	7.8742664e-05
3,307	Cleaning Crowdsourced Labels Using Oracles for Statistical Classification	2019	VLDB	7.4414303e-05
5,061	CDB: A Crowd-Powered Database System	2018	VLDB	6.2912543e-05
5,570	Efficient Algorithms for Crowd-Aided Categorization	2020	VLDB	6.0811968e-05
5,679	Mining an "Anti-Knowledge Base" from Wikipedia Updates with Applications to Fact Checking and Beyond	2020	VLDB	6.0404239e-05
6,397	Human-in-the-loop Data Integration	2017	VLDB	5.7989499e-05
6,428	Domain-Aware Multi-Truth Discovery from Conflicting Sources	2018	VLDB	5.7886234e-05
7,042	Human-in-the-loop Outlier Detection	2020	SIGMOD	5.6142998e-05
7,052	Cost-Effective Data Annotation using Game-Based Crowdsourcing	2019	VLDB	5.6127577e-05
7,500	Crowdsourced Data Management: Overview and Challenges	2017	SIGMOD	5.5083793e-05
9,583	CDB: Optimizing Queries with Crowd-Based Selections and Joins	2017	SIGMOD	5.1571823e-05
9,708	Ground Truth Inference for Weakly Supervised Entity Matching	2023	SIGMOD	5.1374628e-05
10,517	Weighted Set Multi-Cover on Bounded Universe and Applications in Package Recommendation	2026	SIGMOD	4.9793485e-05
10,850	Noisy Interactive Graph Search: An Uncertainty-Based Approach with Online Modeling of Latent Expertise and Difficulty	2026	VLDB	4.9793485e-05
11,907	SA-Q: Observing, Evaluating, and Enhancing the Quality of the Results of Sentiment Analysis Tools	2022	VLDB	4.9793485e-05
12,038	Quality of Sentiment Analysis Tools: The Reasons of Inconsistency	2021	VLDB	4.9793485e-05
12,212	A Rating-Ranking Method for Crowdsourced Top-k Computation	2018	SIGMOD	4.9793485e-05
12,299	Truth Discovery for Spatio-Temporal Events from Crowdsourced Data	2017	VLDB	4.9793485e-05

Outgoing Citations (Sorted by Pagerank)

Showing 16 of 16 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
92	CrowdDB: Answering Queries with Crowdsourcing	2011	SIGMOD	0.00034672523
198	CrowdER: Crowdsourcing Entity Resolution	2012	VLDB	0.00025555196
257	Crowdsourced Databases: Query Processing with People	2011	CIDR	0.00022962347
266	Human-powered Sorts and Joins	2012	VLDB	0.00022739124
749	So Who Won? Dynamic Max Discovery with the Crowd	2012	SIGMOD	0.00014265279
933	Question Selection for Crowd Entity Resolution	2013	VLDB	0.00013004422
987	CrowdScreen: Algorithms for Filtering Data with Humans	2012	SIGMOD	0.00012660627
1,264	CDAS: A Crowdsourcing Data Analytics System	2012	VLDB	0.0001128316
2,272	Resolving Conflicts in Heterogeneous Data by Truth Discovery and Source Reliability Estimation	2014	SIGMOD	8.7121962e-05
3,197	iCrowd: An Adaptive Crowdsourcing Framework	2015	SIGMOD	7.5455503e-05
3,323	A Confidence-Aware Approach for Truth Discovery on Long-Tail Data	2015	VLDB	7.4291935e-05
3,372	Finish Them!: Pricing Algorithms for Human Computation	2014	VLDB	7.3693102e-05
3,839	QASCA: A Quality-Aware Task Assignment System for Crowdsourcing Applications	2015	SIGMOD	6.9921604e-05
4,024	CLAMShell: Speeding up Crowds for Low-latency Data Labeling	2016	VLDB	6.8472018e-05
4,623	Cost-Effective Crowdsourced Entity Resolution: A Partial-Order Approach	2016	SIGMOD	6.4978811e-05
5,024	Crowdsourced Top-k Algorithms: An Experimental Evaluation	2016	VLDB	6.3096186e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	5,024	Crowdsourced Top-k Algorithms: An Experimental Evaluation	2016	VLDB
2	4,623	Cost-Effective Crowdsourced Entity Resolution: A Partial-Order Approach	2016	SIGMOD
3	5,466	Crowdsourcing Applications and Platforms: A Data Management Perspective	2011	VLDB
4	5,250	An Online Cost Sensitive Decision-Making Method in Crowdsourcing Systems	2013	SIGMOD
5	7,500	Crowdsourced Data Management: Overview and Challenges	2017	SIGMOD
6	8,518	Minimizing Efforts in Validating Crowd Answers	2015	SIGMOD
7	7,052	Cost-Effective Data Annotation using Game-Based Crowdsourcing	2019	VLDB
8	1,472	Crowdsourcing Algorithms for Entity Resolution	2014	VLDB
9	933	Question Selection for Crowd Entity Resolution	2013	VLDB
10	6,099	Truth Discovery and Crowdsourcing Aggregation: A Unified Perspective	2015	VLDB