Multi-Query Optimization in MapReduce Framework

Summary: Introduces generalized grouping and partial map-output materialization to share scans and shuffle communication across MapReduce jobs. An optimizer jointly partitions batches and selects techniques, outperforming MRShare by up to 107% on Hadoop. (summarized by gpt-5.6-luna on Jul 24 2026)

Paper ID: hd1f9077807b5b5b4
Venue: VLDB
Year: 2014
Pagerank: 7.5234702e-05
Overall Rank: 3,216 | 78.38%
DOI: 10.14778/2732232.2732234

Incoming Non-self Citations Over Time

Authors

1. Guoping Wang (National University of Singapore)
2. Chee-Yong Chan (National University of Singapore)

BibTeX Citation

@article{wang_vldb14,
        title = {{Multi-Query Optimization in MapReduce Framework}},
        author = {Wang, Guoping and Chan, Chee-Yong},
        journal = {PVLDB},
        series = {{VLDB} '14},
        volume = {7},
        number = {3},
        pages = {145--156},
        doi = {10.14778/2732232.2732234},
        url = {https://doi.org/10.14778/2732232.2732234},
        year = {2014}
}

Incoming Citations (Sorted by Pagerank)

Showing 13 of 13 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
1,745	Selecting Subexpressions to Materialize at Datacenter Scale	2018	VLDB	9.7343818e-05
2,412	S4: Top-k Spreadsheet-Style Search for Query Discovery	2015	SIGMOD	8.5101348e-05
3,033	Chi: A Scalable and Programmable Control Plane for Distributed Stream Processing Systems	2018	VLDB	7.7337998e-05
3,066	Scalable Big Graph Processing in MapReduce	2014	SIGMOD	7.6877117e-05
3,545	Computation Reuse in Analytics Job Service at Microsoft	2018	SIGMOD	7.2134803e-05
5,639	Revisiting Reuse in Main Memory Database Systems	2017	SIGMOD	6.0558642e-05
6,217	Materialization and Reuse Optimizations for Production Data Science Pipelines	2022	SIGMOD	5.8474357e-05
7,955	Distributed Outlier Detection using Compressive Sensing	2015	SIGMOD	5.4185826e-05
8,968	Pipemizer: An Optimizer for Analytics Data Pipelines	2022	VLDB	5.2471751e-05
11,431	Mayura: Exploiting Similarities in Motifs for Temporal Co-Mining	2025	VLDB	4.9793485e-05
12,008	Robust Voice Querying with MUVE: Optimally Visualizing Results of Phonetically Similar Queries	2021	VLDB	4.9793485e-05
12,375	Parallel Evaluation of Multi-Semi-Joins	2016	VLDB	4.9793485e-05
12,447	Shared Execution of Recurring Workloads in MapReduce	2015	VLDB	4.9793485e-05

Outgoing Citations (Sorted by Pagerank)

Showing 10 of 10 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
6	Pig Latin: A Not-So-Foreign Language for Data Processing	2008	SIGMOD	0.001052036
31	Hive - A Warehousing Solution Over a Map-Reduce Framework	2009	VLDB	0.00049839909
651	Building a High-Level Dataflow System on top of Map-Reduce: The Pig Experience	2009	VLDB	0.00015130782
753	Profiling, What-if Analysis, and Cost-based Optimization of MapReduce Programs	2011	VLDB	0.00014237583
823	MRShare: Sharing Across Multiple Queries in MapReduce	2010	VLDB	0.00013648332
1,281	Automatic Optimization for MapReduce Programs	2011	VLDB	0.00011213384
1,924	ReStore: Reusing Results of MapReduce Jobs	2012	VLDB	9.3687009e-05
2,527	Stubby: A Transformation-based Optimizer for MapReduce Workflows	2012	VLDB	8.3425785e-05
5,652	Building Wavelet Histograms on Large Data in MapReduce	2012	VLDB	6.0515502e-05
8,663	MapReduce Programming and Cost-based Optimization? Crossing this Chasm with Starfish	2011	VLDB	5.2928264e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	2,282	A Platform for Scalable One-Pass Analytics using MapReduce	2011	SIGMOD
2	1,281	Automatic Optimization for MapReduce Programs	2011	VLDB
3	43	A Comparison of Approaches to Large-Scale Data Analysis	2009	SIGMOD
4	7,308	Optimization for iterative queries on MapReduce	2014	VLDB
5	753	Profiling, What-if Analysis, and Cost-based Optimization of MapReduce Programs	2011	VLDB
6	1,466	The Performance of MapReduce: An In-depth Study	2010	VLDB
7	75	Map-Reduce-Merge: Simplified Relational Data Processing on Large Clusters	2007	SIGMOD
8	2,931	Efficient Multi-way Theta-Join Processing Using MapReduce	2012	VLDB
9	2,573	Minimal MapReduce Algorithms	2013	SIGMOD
10	823	MRShare: Sharing Across Multiple Queries in MapReduce	2010	VLDB