DBScholar

Back to papers

Profiling, What-if Analysis, and Cost-based Optimization of MapReduce Programs

Summary: First cost-based optimizer for MapReduce programs, tackling the large configuration parameter space with black-box map/reduce functions. Profiler for unmodified programs and a what-if cost estimator enable data-driven optimization; prototype on Hadoop, with cross-domain evaluation. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID
h438bb6896630449c
Venue
VLDB
Year
2011
Pagerank
0.00014194746
Overall Rank
759 | 94.92%
DOI
10.14778/3402703.3402707

Incoming Non-self Citations Over Time

Authors

BibTeX Citation

@article{herodotou_vldb11,
        title = {{Profiling, What-if Analysis, and Cost-based Optimization of MapReduce Programs}},
        author = {Herodotou, Herodotos and Babu, Shivnath},
        journal = {PVLDB},
        series = {{VLDB} '11},
        volume = {4},
        number = {11},
        pages = {1111--1122},
        doi = {10.14778/3402703.3402707},
        url = {https://doi.org/10.14778/3402703.3402707},
        year = {2011}
}

Incoming Citations (Sorted by Pagerank)

Showing 37 of 37 citing papers.

Rank Citing Paper Year Venue Pagerank
676 Robust Estimation of Resource Consumption for SQL Queries using Statistical Techniques 2012 VLDB 0.00014838681
1,074 Interactive Analytical Processing in Big Data Systems: A Cross-Industry Study of MapReduce Workloads 2012 VLDB 0.00012131641
1,083 Hybrid Parallelization Strategies for Large-Scale Machine Learning in SystemML 2014 VLDB 0.00012089325
1,763 Split Query Processing in Polybase 2013 SIGMOD 9.693414e-05
2,011 Pregelix: Big(ger) Graph Analytics on A Dataflow Engine 2015 VLDB 9.1800407e-05
2,124 Opening the Black Boxes in Data Flow Optimization 2012 VLDB 8.9915327e-05
2,250 Cumulon: Optimizing Statistical Data Analysis in the Cloud 2013 SIGMOD 8.7477605e-05
2,532 Stubby: A Transformation-based Optimizer for MapReduce Workflows 2012 VLDB 8.3132509e-05
2,561 Minimal MapReduce Algorithms 2013 SIGMOD 8.2743492e-05
3,228 Multi-Query Optimization in MapReduce Framework 2014 VLDB 7.4957341e-05
3,565 WANalytics: Analytics for a Geo-Distributed Data-Intensive World 2015 CIDR 7.183429e-05
4,134 Resource Elasticity for Large-Scale Machine Learning 2015 SIGMOD 6.7721426e-05
4,303 Clash of the Titans: MapReduce vs. Spark for Large Scale Data Analytics 2015 VLDB 6.6602891e-05
4,489 Real-time Workload Pattern Analysis for Large-scale Cloud Databases 2023 VLDB 6.5619575e-05
4,577 HYPER: Hypothetical Reasoning With What-If and How-To Queries Using a Probabilistic Causal Approach 2022 SIGMOD 6.5111757e-05
5,297 Only Aggressive Elephants are Fast Elephants 2012 VLDB 6.1773472e-05
5,394 PREDIcT: Towards Predicting the Runtime of Large Scale Iterative Analytics 2013 VLDB 6.1351004e-05
5,979 Speedup Your Analytics: Automatic Parameter Tuning for Databases and Big Data Systems 2019 VLDB 5.910201e-05
6,315 Memory-Aware Framework for Efficient Second-Order Random Walk on Large Graphs 2020 SIGMOD 5.7968933e-05
6,718 Toward Interpretable and Actionable Data Analysis with Explanations and Causality 2022 VLDB 5.6843134e-05
6,808 Hadoop's Adolescence: An analysis of Hadoop usage in scientific workloads 2013 VLDB 5.6608074e-05
7,108 KEA: Tuning an Exabyte-Scale Data Infrastructure 2021 SIGMOD 5.5817869e-05
7,284 Submodularity of Distributed Join Computation 2018 SIGMOD 5.5505534e-05
7,492 Fast Multi-Column Sorting in Main-Memory Column-Stores 2016 SIGMOD 5.4981253e-05
7,653 MRTuner: A Toolkit to Enable Holistic Optimization for MapReduce Jobs 2014 VLDB 5.4592529e-05
8,689 MapReduce Programming and Cost-based Optimization? Crossing this Chasm with Starfish 2011 VLDB 5.2728565e-05
9,117 Hyperspace: The Indexing Subsystem of Azure Synapse 2021 VLDB 5.208561e-05
9,288 QMapper for Smart Grid: Migrating SQL-based Application to Hive 2015 SIGMOD 5.1860131e-05
9,728 Efficient Big Data Processing in Hadoop MapReduce 2012 VLDB 5.1205326e-05
9,857 Supporting Scalable Analytics with Latency Constraints 2015 VLDB 5.1064323e-05
10,127 UDAO: A Next-Generation Unified Data Analytics Optimizer 2019 VLDB 5.0638787e-05
11,925 PACk: An Efficient Partition-based Distributed Agglomerative Hierarchical Clustering Algorithm for Deduplication 2022 VLDB 4.9605343e-05
12,180 Automated Performance Management for the Big Data Stack 2019 CIDR 4.9605343e-05
12,212 Cost-Effective, Workload-Adaptive Migration of Big Data Applications to the Cloud 2019 SIGMOD 4.9605343e-05
12,238 An Experimental Evaluation of Garbage Collectors on Big Data Applications 2019 VLDB 4.9605343e-05
12,597 Workload Management for Big Data Analytics 2013 SIGMOD 4.9605343e-05
12,638 Optimization Strategies for A/B Testing on HADOOP 2013 VLDB 4.9605343e-05
Previous Page 1 / 1 Next

Outgoing Citations (Sorted by Pagerank)

Showing 8 of 8 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank Cited Paper Year Venue Pagerank
120 HadoopDB: An Architectural Hybrid of MapReduce and DBMS Technologies for Analytical Workloads 2009 VLDB 0.00031017868
385 HaLoop: Efficient Iterative Data Processing on Large Clusters 2010 VLDB 0.00019419661
677 Hadoop++: Making a Yellow Elephant Run Like a Cheetah (Without It Even Noticing) 2010 VLDB 0.00014835154
799 A Comparison of Join Algorithms for Log Processing in MapReduce 2010 SIGMOD 0.00013839554
825 MRShare: Sharing Across Multiple Queries in MapReduce 2010 VLDB 0.00013600553
944 Starfish: A Self-tuning System for Big Data Analytics 2011 CIDR 0.00012927479
1,286 Automatic Optimization for MapReduce Programs 2011 VLDB 0.00011187304
1,470 The Performance of MapReduce: An In-depth Study 2010 VLDB 0.00010532106
Previous Page 1 / 1 Next

Semantically Similar Papers

# Overall Rank Paper Year Venue
1 120 HadoopDB: An Architectural Hybrid of MapReduce and DBMS Technologies for Analytical Workloads 2009 VLDB
2 76 Map-Reduce-Merge: Simplified Relational Data Processing on Large Clusters 2007 SIGMOD
3 9,728 Efficient Big Data Processing in Hadoop MapReduce 2012 VLDB
4 1,074 Interactive Analytical Processing in Big Data Systems: A Cross-Industry Study of MapReduce Workloads 2012 VLDB
5 2,561 Minimal MapReduce Algorithms 2013 SIGMOD
6 2,299 A Platform for Scalable One-Pass Analytics using MapReduce 2011 SIGMOD
7 1,470 The Performance of MapReduce: An In-depth Study 2010 VLDB
8 3,228 Multi-Query Optimization in MapReduce Framework 2014 VLDB
9 1,286 Automatic Optimization for MapReduce Programs 2011 VLDB
10 8,689 MapReduce Programming and Cost-based Optimization? Crossing this Chasm with Starfish 2011 VLDB