Minimizing Efforts in Validating Crowd Answers

Summary: Proposes a probabilistic model to identify the most beneficial validation questions in crowdsourced answers, enhancing correctness and flagging faulty workers. Demonstrates up to 50% expert savings and near-perfect correctness after only ~20% of questions are validated. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: 5013
Venue: SIGMOD
Year: 2015
Pagerank: 5.4454221e-05
Overall Rank: 8,351 | 42.71%
DOI: 10.1145/2723372.2723731

Incoming Non-self Citations Over Time

Authors

1. Nguyen Quoc Viet Hung (EPFL)
2. Duong Chi Thang (EPFL)
3. Matthias Weidlich (Imperial College London)
4. Karl Aberer (EPFL)

BibTeX Citation

@inproceedings{hung_sigmod15,
        title = {{Minimizing Efforts in Validating Crowd Answers}},
        author = {Hung, Nguyen Quoc Viet and Thang, Duong Chi and Weidlich, Matthias and Aberer, Karl},
        series = {{SIGMOD} '15},
        booktitle = {Proceedings of the {ACM} {SIGMOD} International Conference on Management of Data},
        publisher = {Association for Computing Machinery},
        doi = {10.1145/2723372.2723731},
        url = {https://dl.acm.org/doi/10.1145/2723372.2723731},
        year = {2015}
}

Incoming Citations (Sorted by Pagerank)

Showing 3 of 3 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
3,281	Cleaning Crowdsourced Labels Using Oracles for Statistical Classification	2019	VLDB	7.5706653e-05
7,628	User Guidance for Efficient Fact Checking	2019	VLDB	5.5798467e-05
11,975	Staging User Feedback toward Rapid Conflict Resolution in Data Fusion	2017	SIGMOD	5.093636e-05

Outgoing Citations (Sorted by Pagerank)

Showing 14 of 14 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
39	Efficiently Updating Materialized Views	1986	SIGMOD	0.00047309646
439	Corleone: Hands-Off Crowdsourcing for Entity Matching	2014	SIGMOD	0.00018464913
504	A Bayesian Approach to Discovering Truth from Conflicting Sources for Data Integration	2012	VLDB	0.00017300628
528	On Active Learning of Record Matching Packages	2010	SIGMOD	0.00017100838
714	Guided Data Repair	2011	VLDB	0.00014662041
866	Pay-as-you-go User Feedback for Dataspace Systems	2008	SIGMOD	0.00013519288
1,270	CDAS: A Crowdsourcing Data Analytics System	2012	VLDB	0.00011388453
1,393	Truth Discovery and Copying Detection in a Dynamic World	2009	VLDB	0.00010934682
1,783	Data Fusion – Resolving Data Conflicts for Integration	2009	VLDB	9.772895e-05
2,219	Chimera: Large-Scale Classification using Machine Learning, Rules, and Crowdsourcing	2014	VLDB	8.9303727e-05
2,789	Crowd Mining	2013	SIGMOD	8.1203892e-05
4,153	Whom to Ask? Jury Selection for Decision Making Tasks on Micro-blog Services	2012	VLDB	6.8676997e-05
4,539	Reducing Uncertainty of Schema Matching via Crowdsourcing	2013	VLDB	6.6403822e-05
7,699	SOLOMON: Seeking the Truth Via Copying Detection	2010	VLDB	5.5652613e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	2,626	Scaling Up Crowd-Sourcing to Very Large Datasets: A Case for Active Learning	2015	VLDB
2	3,262	iCrowd: An Adaptive Crowdsourcing Framework	2015	SIGMOD
3	5,813	Efficient Algorithms for Crowd-Aided Categorization	2020	VLDB
4	997	CrowdScreen: Algorithms for Filtering Data with Humans	2012	SIGMOD
5	3,231	Truth Inference in Crowdsourcing: Is the Problem Solved?	2017	VLDB
6	3,914	Optimal Crowd-Powered Rating and Filtering Algorithms	2014	VLDB
7	933	Question Selection for Crowd Entity Resolution	2013	VLDB
8	7,665	Towards Globally Optimal Crowdsourcing Quality Management: The Uniform Worker Setting	2016	SIGMOD
9	5,143	An Online Cost Sensitive Decision-Making Method in Crowdsourcing Systems	2013	SIGMOD
10	12,101	The Importance of Being Expert: Efficient Max-Finding in Crowdsourcing	2015	SIGMOD