DBScholar

Back to papers

Profiling, What-if Analysis, and Cost-based Optimization of MapReduce Programs

Summary: First cost-based optimizer for MapReduce programs, tackling the large configuration parameter space with black-box map/reduce functions. Profiler for unmodified programs and a what-if cost estimator enable data-driven optimization; prototype on Hadoop, with cross-domain evaluation. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID
h438bb6896630449c
Venue
VLDB
Year
2011
Pagerank
0.00014237583
Overall Rank
753 | 94.94%
DOI
10.14778/3402703.3402707

Incoming Non-self Citations Over Time

Authors

BibTeX Citation

@article{herodotou_vldb11,
        title = {{Profiling, What-if Analysis, and Cost-based Optimization of MapReduce Programs}},
        author = {Herodotou, Herodotos and Babu, Shivnath},
        journal = {PVLDB},
        series = {{VLDB} '11},
        volume = {4},
        number = {11},
        pages = {1111--1122},
        doi = {10.14778/3402703.3402707},
        url = {https://doi.org/10.14778/3402703.3402707},
        year = {2011}
}

Incoming Citations (Sorted by Pagerank)

Showing 37 of 37 citing papers.

Rank Citing Paper Year Venue Pagerank
682 Robust Estimation of Resource Consumption for SQL Queries using Statistical Techniques 2012 VLDB 0.0001481781
1,072 Interactive Analytical Processing in Big Data Systems: A Cross-Industry Study of MapReduce Workloads 2012 VLDB 0.00012168947
1,081 Hybrid Parallelization Strategies for Large-Scale Machine Learning in SystemML 2014 VLDB 0.00012123917
1,751 Split Query Processing in Polybase 2013 SIGMOD 9.7291888e-05
1,998 Pregelix: Big(ger) Graph Analytics on A Dataflow Engine 2015 VLDB 9.2144238e-05
2,195 Opening the Black Boxes in Data Flow Optimization 2012 VLDB 8.8781177e-05
2,247 Cumulon: Optimizing Statistical Data Analysis in the Cloud 2013 SIGMOD 8.7585767e-05
2,527 Stubby: A Transformation-based Optimizer for MapReduce Workflows 2012 VLDB 8.3425785e-05
2,573 Minimal MapReduce Algorithms 2013 SIGMOD 8.2821647e-05
3,216 Multi-Query Optimization in MapReduce Framework 2014 VLDB 7.5234702e-05
3,556 WANalytics: Analytics for a Geo-Distributed Data-Intensive World 2015 CIDR 7.2084485e-05
4,116 Resource Elasticity for Large-Scale Machine Learning 2015 SIGMOD 6.7961306e-05
4,288 Clash of the Titans: MapReduce vs. Spark for Large Scale Data Analytics 2015 VLDB 6.6855423e-05
4,468 Real-time Workload Pattern Analysis for Large-scale Cloud Databases 2023 VLDB 6.5863349e-05
4,566 HYPER: Hypothetical Reasoning With What-If and How-To Queries Using a Probabilistic Causal Approach 2022 SIGMOD 6.5310568e-05
5,274 Only Aggressive Elephants are Fast Elephants 2012 VLDB 6.2005706e-05
5,386 PREDIcT: Towards Predicting the Runtime of Large Scale Iterative Analytics 2013 VLDB 6.1522468e-05
5,967 Speedup Your Analytics: Automatic Parameter Tuning for Databases and Big Data Systems 2019 VLDB 5.9324372e-05
6,301 Memory-Aware Framework for Efficient Second-Order Random Walk on Large Graphs 2020 SIGMOD 5.8188746e-05
6,699 Toward Interpretable and Actionable Data Analysis with Explanations and Causality 2022 VLDB 5.7058728e-05
6,786 Hadoop's Adolescence: An analysis of Hadoop usage in scientific workloads 2013 VLDB 5.6820666e-05
7,084 KEA: Tuning an Exabyte-Scale Data Infrastructure 2021 SIGMOD 5.6029455e-05
7,259 Submodularity of Distributed Join Computation 2018 SIGMOD 5.5716049e-05
7,495 Fast Multi-Column Sorting in Main-Memory Column-Stores 2016 SIGMOD 5.5091753e-05
7,632 MRTuner: A Toolkit to Enable Holistic Optimization for MapReduce Jobs 2014 VLDB 5.4793438e-05
8,663 MapReduce Programming and Cost-based Optimization? Crossing this Chasm with Starfish 2011 VLDB 5.2928264e-05
9,088 Hyperspace: The Indexing Subsystem of Azure Synapse 2021 VLDB 5.2283159e-05
9,254 QMapper for Smart Grid: Migrating SQL-based Application to Hive 2015 SIGMOD 5.2056825e-05
9,693 Efficient Big Data Processing in Hadoop MapReduce 2012 VLDB 5.1399537e-05
9,816 Supporting Scalable Analytics with Latency Constraints 2015 VLDB 5.1257999e-05
10,088 UDAO: A Next-Generation Unified Data Analytics Optimizer 2019 VLDB 5.0830849e-05
11,875 PACk: An Efficient Partition-based Distributed Agglomerative Hierarchical Clustering Algorithm for Deduplication 2022 VLDB 4.9793485e-05
12,130 Automated Performance Management for the Big Data Stack 2019 CIDR 4.9793485e-05
12,163 Cost-Effective, Workload-Adaptive Migration of Big Data Applications to the Cloud 2019 SIGMOD 4.9793485e-05
12,189 An Experimental Evaluation of Garbage Collectors on Big Data Applications 2019 VLDB 4.9793485e-05
12,548 Workload Management for Big Data Analytics 2013 SIGMOD 4.9793485e-05
12,589 Optimization Strategies for A/B Testing on HADOOP 2013 VLDB 4.9793485e-05
Previous Page 1 / 1 Next

Outgoing Citations (Sorted by Pagerank)

Showing 8 of 8 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank Cited Paper Year Venue Pagerank
120 HadoopDB: An Architectural Hybrid of MapReduce and DBMS Technologies for Analytical Workloads 2009 VLDB 0.000311132
384 HaLoop: Efficient Iterative Data Processing on Large Clusters 2010 VLDB 0.0001948031
673 Hadoop++: Making a Yellow Elephant Run Like a Cheetah (Without It Even Noticing) 2010 VLDB 0.0001488755
799 A Comparison of Join Algorithms for Log Processing in MapReduce 2010 SIGMOD 0.00013889081
823 MRShare: Sharing Across Multiple Queries in MapReduce 2010 VLDB 0.00013648332
940 Starfish: A Self-tuning System for Big Data Analytics 2011 CIDR 0.00012964445
1,281 Automatic Optimization for MapReduce Programs 2011 VLDB 0.00011213384
1,466 The Performance of MapReduce: An In-depth Study 2010 VLDB 0.00010569837
Previous Page 1 / 1 Next

Semantically Similar Papers

# Overall Rank Paper Year Venue
1 120 HadoopDB: An Architectural Hybrid of MapReduce and DBMS Technologies for Analytical Workloads 2009 VLDB
2 75 Map-Reduce-Merge: Simplified Relational Data Processing on Large Clusters 2007 SIGMOD
3 9,693 Efficient Big Data Processing in Hadoop MapReduce 2012 VLDB
4 1,072 Interactive Analytical Processing in Big Data Systems: A Cross-Industry Study of MapReduce Workloads 2012 VLDB
5 2,573 Minimal MapReduce Algorithms 2013 SIGMOD
6 1,466 The Performance of MapReduce: An In-depth Study 2010 VLDB
7 2,282 A Platform for Scalable One-Pass Analytics using MapReduce 2011 SIGMOD
8 3,216 Multi-Query Optimization in MapReduce Framework 2014 VLDB
9 1,281 Automatic Optimization for MapReduce Programs 2011 VLDB
10 8,663 MapReduce Programming and Cost-based Optimization? Crossing this Chasm with Starfish 2011 VLDB