Truth Inference in Crowdsourcing: Is the Problem Solved?

Summary: Comprehensive survey and benchmark of 17 crowdsourcing truth-inference algorithms on five real datasets, with public code/data. Finds strong dataset dependence and no consistently superior method, showing truth inference remains unsolved and identifying research directions. (summarized by gpt-5.6-luna on Jul 24 2026)

Paper ID: 11738
Venue: VLDB
Year: 2017
Pagerank: 7.6191767e-05
Overall Rank: 3,231 | 77.84%
DOI: 10.14778/3055540.3055547

Incoming Non-self Citations Over Time

Authors

1. Yudian Zheng (University of Hong Kong)
2. Guoliang Li (Tsinghua University)
3. Yuanbing Li (Tsinghua University)
4. Caihua Shan (University of Hong Kong)
5. Reynold Cheng (University of Hong Kong)

BibTeX Citation

@article{zheng_vldb17,
        title = {{Truth Inference in Crowdsourcing: Is the Problem Solved?}},
        author = {Zheng, Yudian and Li, Guoliang and Li, Yuanbing and Shan, Caihua and Cheng, Reynold},
        journal = {PVLDB},
        series = {{VLDB} '17},
        volume = {10},
        number = {5},
        pages = {541--552},
        doi = {10.14778/3055540.3055547},
        url = {https://doi.org/10.14778/3055540.3055547},
        year = {2017}
}

Incoming Citations (Sorted by Pagerank)

Showing 17 of 17 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
2,888	AI Meets Database: AI4DB and DB4AI	2021	SIGMOD	7.9941489e-05
3,281	Cleaning Crowdsourced Labels Using Oracles for Statistical Classification	2019	VLDB	7.5706653e-05
5,211	CDB: A Crowd-Powered Database System	2018	VLDB	6.3161892e-05
5,548	Mining an "Anti-Knowledge Base" from Wikipedia Updates with Applications to Fact Checking and Beyond	2020	VLDB	6.1790655e-05
5,813	Efficient Algorithms for Crowd-Aided Categorization	2020	VLDB	6.0780519e-05
6,787	Domain-Aware Multi-Truth Discovery from Conflicting Sources	2018	VLDB	5.7738798e-05
6,899	Human-in-the-loop Outlier Detection	2020	SIGMOD	5.7431609e-05
6,908	Cost-Effective Data Annotation using Game-Based Crowdsourcing	2019	VLDB	5.7415834e-05
7,357	Crowdsourced Data Management: Overview and Challenges	2017	SIGMOD	5.6346837e-05
7,493	Human-in-the-loop Data Integration	2017	VLDB	5.6046905e-05
9,401	CDB: Optimizing Queries with Crowd-Based Selections and Joins	2017	SIGMOD	5.2755515e-05
9,560	Ground Truth Inference for Weakly Supervised Entity Matching	2023	SIGMOD	5.2528121e-05
10,305	Weighted Set Multi-Cover on Bounded Universe and Applications in Package Recommendation	2026	SIGMOD	5.093636e-05
11,599	SA-Q: Observing, Evaluating, and Enhancing the Quality of the Results of Sentiment Analysis Tools	2022	VLDB	5.093636e-05
11,735	Quality of Sentiment Analysis Tools: The Reasons of Inconsistency	2021	VLDB	5.093636e-05
11,912	A Rating-Ranking Method for Crowdsourced Top-k Computation	2018	SIGMOD	5.093636e-05
12,003	Truth Discovery for Spatio-Temporal Events from Crowdsourced Data	2017	VLDB	5.093636e-05

Outgoing Citations (Sorted by Pagerank)

Showing 16 of 16 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
90	CrowdDB: Answering Queries with Crowdsourcing	2011	SIGMOD	0.00034951786
196	CrowdER: Crowdsourcing Entity Resolution	2012	VLDB	0.00025780596
251	Crowdsourced Databases: Query Processing with People	2011	CIDR	0.00023261113
265	Human-powered Sorts and Joins	2012	VLDB	0.00022935368
743	So Who Won? Dynamic Max Discovery with the Crowd	2012	SIGMOD	0.00014421358
933	Question Selection for Crowd Entity Resolution	2013	VLDB	0.00013111293
997	CrowdScreen: Algorithms for Filtering Data with Humans	2012	SIGMOD	0.00012755983
1,270	CDAS: A Crowdsourcing Data Analytics System	2012	VLDB	0.00011388453
2,307	Resolving Conflicts in Heterogeneous Data by Truth Discovery and Source Reliability Estimation	2014	SIGMOD	8.7750499e-05
3,262	iCrowd: An Adaptive Crowdsourcing Framework	2015	SIGMOD	7.5846052e-05
3,314	Finish Them!: Pricing Algorithms for Human Computation	2014	VLDB	7.5279588e-05
3,542	A Confidence-Aware Approach for Truth Discovery on Long-Tail Data	2015	VLDB	7.3279455e-05
3,904	QASCA: A Quality-Aware Task Assignment System for Crowdsourcing Applications	2015	SIGMOD	7.0304212e-05
3,971	CLAMShell: Speeding up Crowds for Low-latency Data Labeling	2016	VLDB	6.9835263e-05
4,538	Cost-Effective Crowdsourced Entity Resolution: A Partial-Order Approach	2016	SIGMOD	6.6404776e-05
4,915	Crowdsourced Top-k Algorithms: An Experimental Evaluation	2016	VLDB	6.4477856e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	4,915	Crowdsourced Top-k Algorithms: An Experimental Evaluation	2016	VLDB
2	4,538	Cost-Effective Crowdsourced Entity Resolution: A Partial-Order Approach	2016	SIGMOD
3	5,343	Crowdsourcing Applications and Platforms: A Data Management Perspective	2011	VLDB
4	5,143	An Online Cost Sensitive Decision-Making Method in Crowdsourcing Systems	2013	SIGMOD
5	7,357	Crowdsourced Data Management: Overview and Challenges	2017	SIGMOD
6	8,351	Minimizing Efforts in Validating Crowd Answers	2015	SIGMOD
7	6,908	Cost-Effective Data Annotation using Game-Based Crowdsourcing	2019	VLDB
8	1,443	Crowdsourcing Algorithms for Entity Resolution	2014	VLDB
9	933	Question Selection for Crowd Entity Resolution	2013	VLDB
10	5,981	Truth Discovery and Crowdsourcing Aggregation: A Unified Perspective	2015	VLDB