Cost-Effective Data Annotation using Game-Based Crowdsourcing

Summary: CrowdGame enables cost-efficient data annotation via game-based crowdsourcing. Two worker cohorts (rule generators and rule refuters) play a minimax game to optimize coverage vs. precision, using Bayesian estimation for rule quality and task selection; demonstrated on entity matching and relation extraction, outperforming SOTA. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: 11987
Venue: VLDB
Year: 2019
Pagerank: 4.9010083e-05
Overall Rank: 6,868 | 52.23%
DOI: 10.14778/3275536.3275541

Incoming Non-self Citations Over Time

Authors

Incoming Citations (Sorted by Pagerank)

Showing 8 of 8 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
4,884	Relational Data Synthesis using Generative Adversarial Networks: A Design Space Exploration	2020	VLDB	5.8540287e-05
5,347	Adaptive Rule Discovery for Labeling Text Data	2021	SIGMOD	5.5560452e-05
6,569	Domain Adaptation for Deep Entity Resolution	2022	SIGMOD	5.0065379e-05
8,343	CrowdGame: A Game-Based Crowdsourcing System for Cost-Effective Data Labeling	2019	SIGMOD	4.5429217e-05
8,406	DADER: Hands-Off Entity Resolution with Domain Adaptation	2022	VLDB	4.5220083e-05
9,683	Hierarchical Entity Resolution using an Oracle	2022	SIGMOD	4.3047774e-05
9,896	Towards Interpretable and Learnable Risk Analysis for Entity Resolution	2020	SIGMOD	4.2600049e-05
10,091	LLM-Powered Interactive Graph Search: A Scalable and Practical Approach	2026	SIGMOD	4.1945683e-05

Outgoing Citations (Sorted by Pagerank)

Showing 20 of 20 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
94	CrowdDB: Answering Queries with Crowdsourcing	2011	SIGMOD	0.00051013264
254	Snorkel: Rapid Training Data Creation with Weak Supervision	2018	VLDB	0.00030540555
263	CrowdER: Crowdsourcing Entity Resolution	2012	VLDB	0.00029862413
643	Corleone: Hands-Off Crowdsourcing for Entity Matching	2014	SIGMOD	0.00018754451
866	Leveraging Transitive Relations for Crowdsourced Joins	2013	SIGMOD	0.00015801196
1,242	Question Selection for Crowd Entity Resolution	2013	VLDB	0.00013096655
1,491	CDAS: A Crowdsourcing Data Analytics System	2012	VLDB	0.00011694982
1,841	Crowdsourcing Algorithms for Entity Resolution	2014	VLDB	0.00010348858
2,123	Demonstration of Qurk: A Query Processor for Human Operators	2011	SIGMOD	9.4945521e-05
2,175	Falcon: Scaling Up Hands-Off Crowdsourced Entity Matching to Build Cloud Services	2017	SIGMOD	9.3644117e-05
2,809	Deco: A System for Declarative Crowdsourcing	2012	VLDB	8.0869896e-05
2,937	Truth Inference in Crowdsourcing: Is the Problem Solved?	2017	VLDB	7.853108e-05
3,263	QASCA: A Quality-Aware Task Assignment System for Crowdsourcing Applications	2015	SIGMOD	7.3097573e-05
3,322	iCrowd: An Adaptive Crowdsourcing Framework	2015	SIGMOD	7.2230626e-05
4,126	Waldo: An Adaptive Human Interface for Crowd Entity Resolution	2017	SIGMOD	6.4314729e-05
4,451	CLAMShell: Speeding up Crowds for Low-latency Data Labeling	2016	VLDB	6.1738675e-05
4,619	Crowd-Based Deduplication: An Adaptive Approach	2015	SIGMOD	6.0444854e-05
5,362	Cost-Effective Crowdsourced Entity Resolution: A Partial-Order Approach	2016	SIGMOD	5.5473503e-05
7,668	Human-in-the-loop Data Integration	2017	VLDB	4.6834075e-05
11,788	CDB: Optimizing Queries with Crowd-Based Selections and Joins	2017	SIGMOD	4.1945683e-05

Semantically Similar Papers

Overall Rank	Paper	Year	Venue	Pagerank
6,689	Efficient Knowledge Graph Accuracy Evaluation	2019	VLDB	4.9623586e-05
3,118	Scaling Up Crowd-Sourcing to Very Large Datasets: A Case for Active Learning	2015	VLDB	7.5379338e-05
4,827	An Online Cost Sensitive Decision-Making Method in Crowdsourcing Systems	2013	SIGMOD	5.8938399e-05
263	CrowdER: Crowdsourcing Entity Resolution	2012	VLDB	0.00029862413
4,416	CrowdMatcher: Crowd-Assisted Schema Matching	2014	SIGMOD	6.2039225e-05
7,117	Crowdsourced Data Management: Overview and Challenges	2017	SIGMOD	4.826509e-05
866	Leveraging Transitive Relations for Crowdsourced Joins	2013	SIGMOD	0.00015801196
2,937	Truth Inference in Crowdsourcing: Is the Problem Solved?	2017	VLDB	7.853108e-05
5,362	Cost-Effective Crowdsourced Entity Resolution: A Partial-Order Approach	2016	SIGMOD	5.5473503e-05
8,343	CrowdGame: A Game-Based Crowdsourcing System for Cost-Effective Data Labeling	2019	SIGMOD	4.5429217e-05