Cost-Effective Data Annotation using Game-Based Crowdsourcing

Summary: CrowdGame enables cost-efficient data annotation via game-based crowdsourcing. Two worker cohorts (rule generators and rule refuters) play a minimax game to optimize coverage vs. precision, using Bayesian estimation for rule quality and task selection; demonstrated on entity matching and relation extraction, outperforming SOTA. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: 11988
Venue: VLDB
Year: 2019
Pagerank: 4.8963037e-05
Overall Rank: 6,873 | 52.24%
DOI: 10.14778/3275536.3275541

Incoming Non-self Citations Over Time

Authors

Incoming Citations (Sorted by Pagerank)

Showing 8 of 8 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
4,885	Relational Data Synthesis using Generative Adversarial Networks: A Design Space Exploration	2020	VLDB	5.8509546e-05
5,358	Adaptive Rule Discovery for Labeling Text Data	2021	SIGMOD	5.5507062e-05
6,570	Domain Adaptation for Deep Entity Resolution	2022	SIGMOD	5.0017341e-05
8,337	CrowdGame: A Game-Based Crowdsourcing System for Cost-Effective Data Labeling	2019	SIGMOD	4.5385651e-05
8,401	DADER: Hands-Off Entity Resolution with Domain Adaptation	2022	VLDB	4.5176737e-05
9,683	Hierarchical Entity Resolution using an Oracle	2022	SIGMOD	4.3006524e-05
9,895	Towards Interpretable and Learnable Risk Analysis for Entity Resolution	2020	SIGMOD	4.2559233e-05
10,091	LLM-Powered Interactive Graph Search: A Scalable and Practical Approach	2026	SIGMOD	4.1905499e-05

Outgoing Citations (Sorted by Pagerank)

Showing 20 of 20 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
94	CrowdDB: Answering Queries with Crowdsourcing	2011	SIGMOD	0.00051273089
252	Snorkel: Rapid Training Data Creation with Weak Supervision	2018	VLDB	0.00030532082
265	CrowdER: Crowdsourcing Entity Resolution	2012	VLDB	0.00029904018
641	Corleone: Hands-Off Crowdsourcing for Entity Matching	2014	SIGMOD	0.00018759417
863	Leveraging Transitive Relations for Crowdsourced Joins	2013	SIGMOD	0.00015793243
1,242	Question Selection for Crowd Entity Resolution	2013	VLDB	0.00013088272
1,491	CDAS: A Crowdsourcing Data Analytics System	2012	VLDB	0.00011685731
1,842	Crowdsourcing Algorithms for Entity Resolution	2014	VLDB	0.00010342022
2,121	Demonstration of Qurk: A Query Processor for Human Operators	2011	SIGMOD	9.4930973e-05
2,176	Falcon: Scaling Up Hands-Off Crowdsourced Entity Matching to Build Cloud Services	2017	SIGMOD	9.3729351e-05
2,814	Deco: A System for Declarative Crowdsourcing	2012	VLDB	8.0807229e-05
2,944	Truth Inference in Crowdsourcing: Is the Problem Solved?	2017	VLDB	7.8457167e-05
3,268	QASCA: A Quality-Aware Task Assignment System for Crowdsourcing Applications	2015	SIGMOD	7.3027561e-05
3,324	iCrowd: An Adaptive Crowdsourcing Framework	2015	SIGMOD	7.2163115e-05
4,112	Waldo: An Adaptive Human Interface for Crowd Entity Resolution	2017	SIGMOD	6.4329055e-05
4,453	CLAMShell: Speeding up Crowds for Low-latency Data Labeling	2016	VLDB	6.1690121e-05
4,617	Crowd-Based Deduplication: An Adaptive Approach	2015	SIGMOD	6.0400801e-05
5,369	Cost-Effective Crowdsourced Entity Resolution: A Partial-Order Approach	2016	SIGMOD	5.5436995e-05
7,667	Human-in-the-loop Data Integration	2017	VLDB	4.6791871e-05
11,796	CDB: Optimizing Queries with Crowd-Based Selections and Joins	2017	SIGMOD	4.1905499e-05

Semantically Similar Papers

Overall Rank	Paper	Year	Venue	Pagerank
6,690	Efficient Knowledge Graph Accuracy Evaluation	2019	VLDB	4.9575975e-05
4,832	An Online Cost Sensitive Decision-Making Method in Crowdsourcing Systems	2013	SIGMOD	5.8883457e-05
3,020	Scaling Up Crowd-Sourcing to Very Large Datasets: A Case for Active Learning	2015	VLDB	7.6987905e-05
265	CrowdER: Crowdsourcing Entity Resolution	2012	VLDB	0.00029904018
4,414	CrowdMatcher: Crowd-Assisted Schema Matching	2014	SIGMOD	6.1975384e-05
7,116	Crowdsourced Data Management: Overview and Challenges	2017	SIGMOD	4.8219732e-05
863	Leveraging Transitive Relations for Crowdsourced Joins	2013	SIGMOD	0.00015793243
2,944	Truth Inference in Crowdsourcing: Is the Problem Solved?	2017	VLDB	7.8457167e-05
5,369	Cost-Effective Crowdsourced Entity Resolution: A Partial-Order Approach	2016	SIGMOD	5.5436995e-05
8,337	CrowdGame: A Game-Based Crowdsourcing System for Cost-Effective Data Labeling	2019	SIGMOD	4.5385651e-05