Multi-Query Optimization in MapReduce Framework

Summary: Introduces generalized grouping and partial map-output materialization to share scans and shuffle communication across MapReduce jobs. An optimizer jointly partitions batches and selects techniques, outperforming MRShare by up to 107% on Hadoop. (summarized by gpt-5.6-luna on Jul 24 2026)

Paper ID: 10993
Venue: VLDB
Year: 2014
Pagerank: 7.6784171e-05
Overall Rank: 3,163 | 78.31%
DOI: 10.14778/2732232.2732234

Incoming Non-self Citations Over Time

Authors

1. Guoping Wang (National University of Singapore)
2. Chee-Yong Chan (National University of Singapore)

BibTeX Citation

@article{wang_vldb14,
        title = {{Multi-Query Optimization in MapReduce Framework}},
        author = {Wang, Guoping and Chan, Chee-Yong},
        journal = {PVLDB},
        series = {{VLDB} '14},
        volume = {7},
        number = {3},
        pages = {145--156},
        doi = {10.14778/2732232.2732234},
        url = {https://doi.org/10.14778/2732232.2732234},
        year = {2014}
}

Incoming Citations (Sorted by Pagerank)

Showing 13 of 13 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
1,765	Selecting Subexpressions to Materialize at Datacenter Scale	2018	VLDB	9.8079546e-05
2,387	S4: Top-k Spreadsheet-Style Search for Query Discovery	2015	SIGMOD	8.6482009e-05
3,008	Scalable Big Graph Processing in MapReduce	2014	SIGMOD	7.8578871e-05
3,117	Chi: A Scalable and Programmable Control Plane for Distributed Stream Processing Systems	2018	VLDB	7.7382559e-05
3,605	Computation Reuse in Analytics Job Service at Microsoft	2018	SIGMOD	7.2640711e-05
5,947	Revisiting Reuse in Main Memory Database Systems	2017	SIGMOD	6.0323707e-05
6,309	Materialization and Reuse Optimizations for Production Data Science Pipelines	2022	SIGMOD	5.9189554e-05
7,795	Distributed Outlier Detection using Compressive Sensing	2015	SIGMOD	5.5429229e-05
8,864	Pipemizer: An Optimizer for Analytics Data Pipelines	2022	VLDB	5.355022e-05
11,074	Mayura: Exploiting Similarities in Motifs for Temporal Co-Mining	2025	VLDB	5.093636e-05
11,704	Robust Voice Querying with MUVE: Optimally Visualizing Results of Phonetically Similar Queries	2021	VLDB	5.093636e-05
12,082	Parallel Evaluation of Multi-Semi-Joins	2016	VLDB	5.093636e-05
12,156	Shared Execution of Recurring Workloads in MapReduce	2015	VLDB	5.093636e-05

Outgoing Citations (Sorted by Pagerank)

Showing 10 of 10 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
6	Pig Latin: A Not-So-Foreign Language for Data Processing	2008	SIGMOD	0.0010686205
32	Hive - A Warehousing Solution Over a Map-Reduce Framework	2009	VLDB	0.00050111008
642	Building a High-Level Dataflow System on top of Map-Reduce: The Pig Experience	2009	VLDB	0.00015395331
735	Profiling, What-if Analysis, and Cost-based Optimization of MapReduce Programs	2011	VLDB	0.00014522606
803	MRShare: Sharing Across Multiple Queries in MapReduce	2010	VLDB	0.00013899943
1,257	Automatic Optimization for MapReduce Programs	2011	VLDB	0.000114432
1,883	ReStore: Reusing Results of MapReduce Jobs	2012	VLDB	9.5421713e-05
2,486	Stubby: A Transformation-based Optimizer for MapReduce Workflows	2012	VLDB	8.5143189e-05
5,522	Building Wavelet Histograms on Large Data in MapReduce	2012	VLDB	6.1871697e-05
8,497	MapReduce Programming and Cost-based Optimization? Crossing this Chasm with Starfish	2011	VLDB	5.41357e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	769	A Comparison of Join Algorithms for Log Processing in MapReduce	2010	SIGMOD
2	1,257	Automatic Optimization for MapReduce Programs	2011	VLDB
3	44	A Comparison of Approaches to Large-Scale Data Analysis	2009	SIGMOD
4	7,168	Optimization for iterative queries on MapReduce	2014	VLDB
5	735	Profiling, What-if Analysis, and Cost-based Optimization of MapReduce Programs	2011	VLDB
6	1,436	The Performance of MapReduce: An In-depth Study	2010	VLDB
7	72	Map-Reduce-Merge: Simplified Relational Data Processing on Large Clusters	2007	SIGMOD
8	2,887	Efficient Multi-way Theta-Join Processing Using MapReduce	2012	VLDB
9	2,539	Minimal MapReduce Algorithms	2013	SIGMOD
10	803	MRShare: Sharing Across Multiple Queries in MapReduce	2010	VLDB