Scaling Up Crowd-Sourcing to Very Large Datasets: A Case for Active Learning

Summary: Scalable active-learning for crowd-sourced databases, combining ML with human labeling via nonparametric bootstrap. MTurk and 15 datasets show 1–2 orders of magnitude fewer questions than baselines and 4.5–44× faster than prior AL. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: hf18fc83ae7e33252
Venue: VLDB
Year: 2015
Pagerank: 8.1565673e-05
Overall Rank: 2,647 | 82.26%
DOI: 10.14778/2735471.2735474
PDF: Download (CC BY-NC-ND 3.0)

Incoming Non-self Citations Over Time

Authors

1. Barzan Mozafari (University of Michigan)
2. Purna Sarkar (University of California Berkeley)
3. Michael Franklin (University of California Berkeley)
4. Michael Jordan (University of California Berkeley)
5. Samuel Madden (Massachusetts Institute of Technology)

BibTeX Citation

@article{mozafari_vldb15,
        title = {{Scaling Up Crowd-Sourcing to Very Large Datasets: A Case for Active Learning}},
        author = {Mozafari, Barzan and Sarkar, Purna and Franklin, Michael and Jordan, Michael and Madden, Samuel},
        journal = {PVLDB},
        series = {{VLDB} '15},
        volume = {8},
        number = {2},
        pages = {125},
        doi = {10.14778/2735471.2735474},
        url = {https://doi.org/10.14778/2735471.2735474},
        year = {2015}
}

Incoming Citations (Sorted by Pagerank)

Showing 18 of 18 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
460	ActiveClean: Interactive Data Cleaning For Statistical Modeling	2016	VLDB	0.00017836918
770	VerdictDB: Universalizing Approximate Query Processing	2018	SIGMOD	0.00014086699
989	Data Cleaning: Overview and Emerging Challenges	2016	SIGMOD	0.00012638176
1,650	Falcon: Scaling Up Hands-Off Crowdsourced Entity Matching to Build Cloud Services	2017	SIGMOD	9.9690948e-05
2,290	Active Learning for ML Enhanced Database Systems	2020	SIGMOD	8.6813363e-05
2,476	A Comprehensive Benchmark Framework for Active Learning Methods in Entity Matching	2020	SIGMOD	8.3932405e-05
3,318	Cleaning Crowdsourced Labels Using Oracles for Statistical Classification	2019	VLDB	7.4142395e-05
4,029	CLAMShell: Speeding up Crowds for Low-latency Data Labeling	2016	VLDB	6.8285702e-05
4,948	Deep Indexed Active Learning for Matching Heterogeneous Entity Representations	2022	VLDB	6.3276067e-05
5,931	In Search of an Entity Resolution OASIS: Optimal Asymptotic Sequential Importance Sampling	2017	VLDB	5.9263018e-05
6,791	Crowdsourced Data Management: Overview and Challenges	2017	SIGMOD	5.6647309e-05
7,780	User Guidance for Efficient Fact Checking	2019	VLDB	5.4377978e-05
9,823	The Battleship Approach to the Low Resource Entity Matching Problem	2023	SIGMOD	5.1089508e-05
10,280	Towards Interpretable and Learnable Risk Analysis for Entity Resolution	2020	SIGMOD	5.0334833e-05
10,819	ALER: An Active Learning Hybrid System for Efficient Entity Resolution	2026	VLDB	4.9605343e-05
11,794	VersaMatch: Ontology Matching with Weak Supervision	2023	VLDB	4.9605343e-05
12,140	Recommending Deployment Strategies for Collaborative Tasks	2020	SIGMOD	4.9605343e-05
12,322	Staging User Feedback toward Rapid Conflict Resolution in Data Fusion	2017	SIGMOD	4.9605343e-05

Outgoing Citations (Sorted by Pagerank)

Showing 8 of 8 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
92	CrowdDB: Answering Queries with Crowdsourcing	2011	SIGMOD	0.00034613457
199	CrowdER: Crowdsourcing Entity Resolution	2012	VLDB	0.0002551523
266	Human-powered Sorts and Joins	2012	VLDB	0.00022701006
535	On Active Learning of Record Matching Packages	2010	SIGMOD	0.00016768308
991	CrowdScreen: Algorithms for Filtering Data with Humans	2012	SIGMOD	0.00012632003
1,924	The Analytical Bootstrap: a New Method for Fast Error Estimation in Approximate Query Processing	2014	SIGMOD	9.3535088e-05
1,932	Counting with the Crowd	2013	VLDB	9.3441665e-05
5,487	ABS: a System for Scalable Approximate Queries with Accuracy Guarantees	2014	SIGMOD	6.0974772e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	1,473	Crowdsourcing Algorithms for Entity Resolution	2014	VLDB
2	8,541	Minimizing Efforts in Validating Crowd Answers	2015	SIGMOD
3	6,791	Crowdsourced Data Management: Overview and Challenges	2017	SIGMOD
4	5,052	Crowdsourced Top-k Algorithms: An Experimental Evaluation	2016	VLDB
5	7,074	Cost-Effective Data Annotation using Game-Based Crowdsourcing	2019	VLDB
6	1,267	CDAS: A Crowdsourcing Data Analytics System	2012	VLDB
7	8,690	CrowdGame: A Game-Based Crowdsourcing System for Cost-Effective Data Labeling	2019	SIGMOD
8	932	Question Selection for Crowd Entity Resolution	2013	VLDB
9	5,586	Efficient Algorithms for Crowd-Aided Categorization	2020	VLDB
10	5,276	An Online Cost Sensitive Decision-Making Method in Crowdsourcing Systems	2013	SIGMOD