Scaling Up Crowd-Sourcing to Very Large Datasets: A Case for Active Learning

Summary: Scalable active-learning for crowd-sourced databases, combining ML with human labeling via nonparametric bootstrap. MTurk and 15 datasets show 1–2 orders of magnitude fewer questions than baselines and 4.5–44× faster than prior AL. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: hf18fc83ae7e33252
Venue: VLDB
Year: 2015
Pagerank: 8.1741373e-05
Overall Rank: 2,643 | 82.24%
DOI: 10.14778/2735471.2735474
PDF: Download (CC BY-NC-ND 3.0)

Incoming Non-self Citations Over Time

Authors

1. Barzan Mozafari (University of Michigan)
2. Purna Sarkar (University of California Berkeley)
3. Michael Franklin (University of California Berkeley)
4. Michael Jordan (University of California Berkeley)
5. Samuel Madden (Massachusetts Institute of Technology)

BibTeX Citation

@article{mozafari_vldb15,
        title = {{Scaling Up Crowd-Sourcing to Very Large Datasets: A Case for Active Learning}},
        author = {Mozafari, Barzan and Sarkar, Purna and Franklin, Michael and Jordan, Michael and Madden, Samuel},
        journal = {PVLDB},
        series = {{VLDB} '15},
        volume = {8},
        number = {2},
        pages = {125},
        doi = {10.14778/2735471.2735474},
        url = {https://doi.org/10.14778/2735471.2735474},
        year = {2015}
}

Incoming Citations (Sorted by Pagerank)

Showing 18 of 18 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
483	ActiveClean: Interactive Data Cleaning For Statistical Modeling	2016	VLDB	0.00017584249
772	VerdictDB: Universalizing Approximate Query Processing	2018	SIGMOD	0.0001409096
1,044	Data Cleaning: Overview and Emerging Challenges	2016	SIGMOD	0.00012329478
1,655	Falcon: Scaling Up Hands-Off Crowdsourced Entity Matching to Build Cloud Services	2017	SIGMOD	9.9694129e-05
2,278	Active Learning for ML Enhanced Database Systems	2020	SIGMOD	8.7057608e-05
2,475	A Comprehensive Benchmark Framework for Active Learning Methods in Entity Matching	2020	SIGMOD	8.4074448e-05
3,308	Cleaning Crowdsourced Labels Using Oracles for Statistical Classification	2019	VLDB	7.4379101e-05
4,025	CLAMShell: Speeding up Crowds for Low-latency Data Labeling	2016	VLDB	6.8439934e-05
4,941	Deep Indexed Active Learning for Matching Heterogeneous Entity Representations	2022	VLDB	6.3428387e-05
5,919	In Search of an Entity Resolution OASIS: Optimal Asymptotic Sequential Importance Sampling	2017	VLDB	5.9459629e-05
7,505	Crowdsourced Data Management: Overview and Challenges	2017	SIGMOD	5.5057737e-05
7,765	User Guidance for Efficient Fact Checking	2019	VLDB	5.4558382e-05
9,791	The Battleship Approach to the Low Resource Entity Matching Problem	2023	SIGMOD	5.1259002e-05
10,250	Towards Interpretable and Learnable Risk Analysis for Entity Resolution	2020	SIGMOD	5.0501823e-05
10,758	ALER: An Active Learning Hybrid System for Efficient Entity Resolution	2026	VLDB	4.9769913e-05
11,750	VersaMatch: Ontology Matching with Weak Supervision	2023	VLDB	4.9769913e-05
12,096	Recommending Deployment Strategies for Collaborative Tasks	2020	SIGMOD	4.9769913e-05
12,279	Staging User Feedback toward Rapid Conflict Resolution in Data Fusion	2017	SIGMOD	4.9769913e-05

Outgoing Citations (Sorted by Pagerank)

Showing 8 of 8 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
92	CrowdDB: Answering Queries with Crowdsourcing	2011	SIGMOD	0.00034670735
198	CrowdER: Crowdsourcing Entity Resolution	2012	VLDB	0.00025546182
266	Human-powered Sorts and Joins	2012	VLDB	0.00022735106
534	On Active Learning of Record Matching Packages	2010	SIGMOD	0.00016799
987	CrowdScreen: Algorithms for Filtering Data with Humans	2012	SIGMOD	0.00012655094
1,916	The Analytical Bootstrap: a New Method for Fast Error Estimation in Approximate Query Processing	2014	SIGMOD	9.3822742e-05
1,926	Counting with the Crowd	2013	VLDB	9.3636304e-05
5,474	ABS: a System for Scalable Approximate Queries with Accuracy Guarantees	2014	SIGMOD	6.1159955e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	8,524	Minimizing Efforts in Validating Crowd Answers	2015	SIGMOD
2	1,472	Crowdsourcing Algorithms for Entity Resolution	2014	VLDB
3	7,505	Crowdsourced Data Management: Overview and Challenges	2017	SIGMOD
4	5,027	Crowdsourced Top-k Algorithms: An Experimental Evaluation	2016	VLDB
5	7,054	Cost-Effective Data Annotation using Game-Based Crowdsourcing	2019	VLDB
6	1,265	CDAS: A Crowdsourcing Data Analytics System	2012	VLDB
7	8,672	CrowdGame: A Game-Based Crowdsourcing System for Cost-Effective Data Labeling	2019	SIGMOD
8	934	Question Selection for Crowd Entity Resolution	2013	VLDB
9	5,572	Efficient Algorithms for Crowd-Aided Categorization	2020	VLDB
10	5,253	An Online Cost Sensitive Decision-Making Method in Crowdsourcing Systems	2013	SIGMOD