Scaling Up Crowd-Sourcing to Very Large Datasets: A Case for Active Learning

Summary: Scalable active-learning for crowd-sourced databases, combining ML with human labeling via nonparametric bootstrap. MTurk and 15 datasets show 1–2 orders of magnitude fewer questions than baselines and 4.5–44× faster than prior AL. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: 11191
Venue: VLDB
Year: 2015
Pagerank: 8.3291889e-05
Overall Rank: 2,626 | 81.99%
DOI: 10.14778/2735471.2735474

Incoming Non-self Citations Over Time

Authors

1. Barzan Mozafari (University of Michigan)
2. Purna Sarkar (University of California Berkeley)
3. Michael Franklin (University of California Berkeley)
4. Michael Jordan (University of California Berkeley)
5. Samuel Madden (Massachusetts Institute of Technology)

BibTeX Citation

@article{mozafari_vldb15,
        title = {{Scaling Up Crowd-Sourcing to Very Large Datasets: A Case for Active Learning}},
        author = {Mozafari, Barzan and Sarkar, Purna and Franklin, Michael and Jordan, Michael and Madden, Samuel},
        journal = {PVLDB},
        series = {{VLDB} '15},
        volume = {8},
        number = {2},
        pages = {125},
        doi = {10.14778/2735471.2735474},
        url = {https://doi.org/10.14778/2735471.2735474},
        year = {2015}
}

Incoming Citations (Sorted by Pagerank)

Showing 18 of 18 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
582	ActiveClean: Interactive Data Cleaning For Statistical Modeling	2016	VLDB	0.00016148948
772	VerdictDB: Universalizing Approximate Query Processing	2018	SIGMOD	0.00014147905
1,323	Data Cleaning: Overview and Emerging Challenges	2016	SIGMOD	0.00011152602
1,643	Falcon: Scaling Up Hands-Off Crowdsourced Entity Matching to Build Cloud Services	2017	SIGMOD	0.00010134956
2,313	Active Learning for ML Enhanced Database Systems	2020	SIGMOD	8.762627e-05
2,463	A Comprehensive Benchmark Framework for Active Learning Methods in Entity Matching	2020	SIGMOD	8.5486912e-05
3,281	Cleaning Crowdsourced Labels Using Oracles for Statistical Classification	2019	VLDB	7.5706653e-05
3,971	CLAMShell: Speeding up Crowds for Low-latency Data Labeling	2016	VLDB	6.9835263e-05
4,824	Deep Indexed Active Learning for Matching Heterogeneous Entity Representations	2022	VLDB	6.4914944e-05
5,899	In Search of an Entity Resolution OASIS: Optimal Asymptotic Sequential Importance Sampling	2017	VLDB	6.0469241e-05
7,357	Crowdsourced Data Management: Overview and Challenges	2017	SIGMOD	5.6346837e-05
7,628	User Guidance for Efficient Fact Checking	2019	VLDB	5.5798467e-05
9,610	The Battleship Approach to the Low Resource Entity Matching Problem	2023	SIGMOD	5.2460348e-05
10,049	Towards Interpretable and Learnable Risk Analysis for Entity Resolution	2020	SIGMOD	5.1685424e-05
10,566	ALER: An Active Learning Hybrid System for Efficient Entity Resolution	2026	VLDB	5.093636e-05
11,430	VersaMatch: Ontology Matching with Weak Supervision	2023	VLDB	5.093636e-05
11,788	Recommending Deployment Strategies for Collaborative Tasks	2020	SIGMOD	5.093636e-05
11,975	Staging User Feedback toward Rapid Conflict Resolution in Data Fusion	2017	SIGMOD	5.093636e-05

Outgoing Citations (Sorted by Pagerank)

Showing 8 of 8 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
90	CrowdDB: Answering Queries with Crowdsourcing	2011	SIGMOD	0.00034951786
196	CrowdER: Crowdsourcing Entity Resolution	2012	VLDB	0.00025780596
265	Human-powered Sorts and Joins	2012	VLDB	0.00022935368
528	On Active Learning of Record Matching Packages	2010	SIGMOD	0.00017100838
997	CrowdScreen: Algorithms for Filtering Data with Humans	2012	SIGMOD	0.00012755983
1,872	The Analytical Bootstrap: a New Method for Fast Error Estimation in Approximate Query Processing	2014	SIGMOD	9.5759874e-05
1,910	Counting with the Crowd	2013	VLDB	9.4972788e-05
5,358	ABS: a System for Scalable Approximate Queries with Accuracy Guarantees	2014	SIGMOD	6.2492955e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	1,443	Crowdsourcing Algorithms for Entity Resolution	2014	VLDB
2	8,351	Minimizing Efforts in Validating Crowd Answers	2015	SIGMOD
3	7,357	Crowdsourced Data Management: Overview and Challenges	2017	SIGMOD
4	4,915	Crowdsourced Top-k Algorithms: An Experimental Evaluation	2016	VLDB
5	6,908	Cost-Effective Data Annotation using Game-Based Crowdsourcing	2019	VLDB
6	1,270	CDAS: A Crowdsourcing Data Analytics System	2012	VLDB
7	8,495	CrowdGame: A Game-Based Crowdsourcing System for Cost-Effective Data Labeling	2019	SIGMOD
8	933	Question Selection for Crowd Entity Resolution	2013	VLDB
9	5,813	Efficient Algorithms for Crowd-Aided Categorization	2020	VLDB
10	5,143	An Online Cost Sensitive Decision-Making Method in Crowdsourcing Systems	2013	SIGMOD