Truth Inference in Crowdsourcing: Is the Problem Solved?

Summary: Comprehensive survey and benchmark of 17 crowdsourcing truth-inference algorithms on five real datasets, with public code/data. Finds strong dataset dependence and no consistently superior method, showing truth inference remains unsolved and identifying research directions. (summarized by gpt-5.6-luna on Jul 24 2026)

Paper ID: h35033f515c359b11
Venue: VLDB
Year: 2017
Pagerank: 7.7177453e-05
Overall Rank: 3,028 | 79.71%
DOI: 10.14778/3055540.3055547
PDF: Download (CC BY-NC-ND 4.0)

Incoming Non-self Citations Over Time

Authors

1. Yudian Zheng (University of Hong Kong)
2. Guoliang Li (Tsinghua University)
3. Yuanbing Li (Tsinghua University)
4. Caihua Shan (University of Hong Kong)
5. Reynold Cheng (University of Hong Kong)

BibTeX Citation

@article{zheng_vldb17,
        title = {{Truth Inference in Crowdsourcing: Is the Problem Solved?}},
        author = {Zheng, Yudian and Li, Guoliang and Li, Yuanbing and Shan, Caihua and Cheng, Reynold},
        journal = {PVLDB},
        series = {{VLDB} '17},
        volume = {10},
        number = {5},
        pages = {541--552},
        doi = {10.14778/3055540.3055547},
        url = {https://doi.org/10.14778/3055540.3055547},
        year = {2017}
}

Incoming Citations (Sorted by Pagerank)

Showing 19 of 19 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
2,913	AI Meets Database: AI4DB and DB4AI	2021	SIGMOD	7.855786e-05
3,318	Cleaning Crowdsourced Labels Using Oracles for Statistical Classification	2019	VLDB	7.4142395e-05
5,051	CDB: A Crowd-Powered Database System	2018	VLDB	6.2863221e-05
5,586	Efficient Algorithms for Crowd-Aided Categorization	2020	VLDB	6.0582193e-05
5,694	Mining an "Anti-Knowledge Base" from Wikipedia Updates with Applications to Fact Checking and Beyond	2020	VLDB	6.0176005e-05
6,404	Human-in-the-loop Data Integration	2017	VLDB	5.7797097e-05
6,442	Domain-Aware Multi-Truth Discovery from Conflicting Sources	2018	VLDB	5.7667567e-05
6,791	Crowdsourced Data Management: Overview and Challenges	2017	SIGMOD	5.6647309e-05
7,064	Human-in-the-loop Outlier Detection	2020	SIGMOD	5.5930864e-05
7,074	Cost-Effective Data Annotation using Game-Based Crowdsourcing	2019	VLDB	5.5915502e-05
9,617	CDB: Optimizing Queries with Crowd-Based Selections and Joins	2017	SIGMOD	5.1376962e-05
9,744	Ground Truth Inference for Weakly Supervised Entity Matching	2023	SIGMOD	5.1180929e-05
10,385	Budgeted Interactive Property Graph Repair with GNN	2027	SIGMOD	4.9605343e-05
10,594	Weighted Set Multi-Cover on Bounded Universe and Applications in Package Recommendation	2026	SIGMOD	4.9605343e-05
10,920	Noisy Interactive Graph Search: An Uncertainty-Based Approach with Online Modeling of Latent Expertise and Difficulty	2026	VLDB	4.9605343e-05
11,957	SA-Q: Observing, Evaluating, and Enhancing the Quality of the Results of Sentiment Analysis Tools	2022	VLDB	4.9605343e-05
12,088	Quality of Sentiment Analysis Tools: The Reasons of Inconsistency	2021	VLDB	4.9605343e-05
12,261	A Rating-Ranking Method for Crowdsourced Top-k Computation	2018	SIGMOD	4.9605343e-05
12,348	Truth Discovery for Spatio-Temporal Events from Crowdsourced Data	2017	VLDB	4.9605343e-05

Outgoing Citations (Sorted by Pagerank)

Showing 16 of 16 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
92	CrowdDB: Answering Queries with Crowdsourcing	2011	SIGMOD	0.00034613457
199	CrowdER: Crowdsourcing Entity Resolution	2012	VLDB	0.0002551523
257	Crowdsourced Databases: Query Processing with People	2011	CIDR	0.00022914527
266	Human-powered Sorts and Joins	2012	VLDB	0.00022701006
754	So Who Won? Dynamic Max Discovery with the Crowd	2012	SIGMOD	0.00014226032
932	Question Selection for Crowd Entity Resolution	2013	VLDB	0.00012981731
991	CrowdScreen: Algorithms for Filtering Data with Humans	2012	SIGMOD	0.00012632003
1,267	CDAS: A Crowdsourcing Data Analytics System	2012	VLDB	0.00011253716
2,283	Resolving Conflicts in Heterogeneous Data by Truth Discovery and Source Reliability Estimation	2014	SIGMOD	8.6888288e-05
3,210	iCrowd: An Adaptive Crowdsourcing Framework	2015	SIGMOD	7.5170875e-05
3,327	A Confidence-Aware Approach for Truth Discovery on Long-Tail Data	2015	VLDB	7.4101504e-05
3,368	Finish Them!: Pricing Algorithms for Human Computation	2014	VLDB	7.351092e-05
3,849	QASCA: A Quality-Aware Task Assignment System for Crowdsourcing Applications	2015	SIGMOD	6.9657571e-05
4,029	CLAMShell: Speeding up Crowds for Low-latency Data Labeling	2016	VLDB	6.8285702e-05
4,626	Cost-Effective Crowdsourced Entity Resolution: A Partial-Order Approach	2016	SIGMOD	6.480594e-05
5,052	Crowdsourced Top-k Algorithms: An Experimental Evaluation	2016	VLDB	6.2857835e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	5,052	Crowdsourced Top-k Algorithms: An Experimental Evaluation	2016	VLDB
2	4,626	Cost-Effective Crowdsourced Entity Resolution: A Partial-Order Approach	2016	SIGMOD
3	5,478	Crowdsourcing Applications and Platforms: A Data Management Perspective	2011	VLDB
4	5,276	An Online Cost Sensitive Decision-Making Method in Crowdsourcing Systems	2013	SIGMOD
5	6,791	Crowdsourced Data Management: Overview and Challenges	2017	SIGMOD
6	8,541	Minimizing Efforts in Validating Crowd Answers	2015	SIGMOD
7	7,074	Cost-Effective Data Annotation using Game-Based Crowdsourcing	2019	VLDB
8	1,473	Crowdsourcing Algorithms for Entity Resolution	2014	VLDB
9	932	Question Selection for Crowd Entity Resolution	2013	VLDB
10	6,110	Truth Discovery and Crowdsourcing Aggregation: A Unified Perspective	2015	VLDB