DBScholar

Back to papers

Profiling, What-if Analysis, and Cost-based Optimization of MapReduce Programs

Summary: First cost-based optimizer for MapReduce programs, tackling the large configuration parameter space with black-box map/reduce functions. Profiler for unmodified programs and a what-if cost estimator enable data-driven optimization; prototype on Hadoop, with cross-domain evaluation. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID
h438bb6896630449c
Venue
VLDB
Year
2011
Pagerank
0.00014231311
Overall Rank
754 | 94.94%
DOI
10.14778/3402703.3402707

Incoming Non-self Citations Over Time

Authors

BibTeX Citation

@article{herodotou_vldb11,
        title = {{Profiling, What-if Analysis, and Cost-based Optimization of MapReduce Programs}},
        author = {Herodotou, Herodotos and Babu, Shivnath},
        journal = {PVLDB},
        series = {{VLDB} '11},
        volume = {4},
        number = {11},
        pages = {1111--1122},
        doi = {10.14778/3402703.3402707},
        url = {https://doi.org/10.14778/3402703.3402707},
        year = {2011}
}

Incoming Citations (Sorted by Pagerank)

Showing 37 of 37 citing papers.

Rank Citing Paper Year Venue Pagerank
682 Robust Estimation of Resource Consumption for SQL Queries using Statistical Techniques 2012 VLDB 0.00014814858
1,073 Interactive Analytical Processing in Big Data Systems: A Cross-Industry Study of MapReduce Workloads 2012 VLDB 0.00012163258
1,082 Hybrid Parallelization Strategies for Large-Scale Machine Learning in SystemML 2014 VLDB 0.00012118261
1,752 Split Query Processing in Polybase 2013 SIGMOD 9.7246277e-05
2,000 Pregelix: Big(ger) Graph Analytics on A Dataflow Engine 2015 VLDB 9.2101691e-05
2,197 Opening the Black Boxes in Data Flow Optimization 2012 VLDB 8.8740089e-05
2,249 Cumulon: Optimizing Statistical Data Analysis in the Cloud 2013 SIGMOD 8.7544468e-05
2,527 Stubby: A Transformation-based Optimizer for MapReduce Workflows 2012 VLDB 8.3386546e-05
2,573 Minimal MapReduce Algorithms 2013 SIGMOD 8.2782871e-05
3,217 Multi-Query Optimization in MapReduce Framework 2014 VLDB 7.5199872e-05
3,558 WANalytics: Analytics for a Geo-Distributed Data-Intensive World 2015 CIDR 7.2050361e-05
4,117 Resource Elasticity for Large-Scale Machine Learning 2015 SIGMOD 6.7929814e-05
4,288 Clash of the Titans: MapReduce vs. Spark for Large Scale Data Analytics 2015 VLDB 6.6823776e-05
4,470 Real-time Workload Pattern Analysis for Large-scale Cloud Databases 2023 VLDB 6.5833414e-05
4,568 HYPER: Hypothetical Reasoning With What-If and How-To Queries Using a Probabilistic Causal Approach 2022 SIGMOD 6.5279657e-05
5,278 Only Aggressive Elephants are Fast Elephants 2012 VLDB 6.1976381e-05
5,392 PREDIcT: Towards Predicting the Runtime of Large Scale Iterative Analytics 2013 VLDB 6.1494698e-05
5,969 Speedup Your Analytics: Automatic Parameter Tuning for Databases and Big Data Systems 2019 VLDB 5.9296488e-05
6,305 Memory-Aware Framework for Efficient Second-Order Random Walk on Large Graphs 2020 SIGMOD 5.8161222e-05
6,703 Toward Interpretable and Actionable Data Analysis with Explanations and Causality 2022 VLDB 5.7031717e-05
6,792 Hadoop's Adolescence: An analysis of Hadoop usage in scientific workloads 2013 VLDB 5.6794384e-05
7,086 KEA: Tuning an Exabyte-Scale Data Infrastructure 2021 SIGMOD 5.6002958e-05
7,266 Submodularity of Distributed Join Computation 2018 SIGMOD 5.5689674e-05
7,473 Fast Multi-Column Sorting in Main-Memory Column-Stores 2016 SIGMOD 5.5162245e-05
7,638 MRTuner: A Toolkit to Enable Holistic Optimization for MapReduce Jobs 2014 VLDB 5.4767699e-05
8,671 MapReduce Programming and Cost-based Optimization? Crossing this Chasm with Starfish 2011 VLDB 5.2903241e-05
9,098 Hyperspace: The Indexing Subsystem of Azure Synapse 2021 VLDB 5.2258409e-05
9,264 QMapper for Smart Grid: Migrating SQL-based Application to Hive 2015 SIGMOD 5.2032182e-05
9,699 Efficient Big Data Processing in Hadoop MapReduce 2012 VLDB 5.1375205e-05
9,823 Supporting Scalable Analytics with Latency Constraints 2015 VLDB 5.1233734e-05
10,093 UDAO: A Next-Generation Unified Data Analytics Optimizer 2019 VLDB 5.0806786e-05
11,881 PACk: An Efficient Partition-based Distributed Agglomerative Hierarchical Clustering Algorithm for Deduplication 2022 VLDB 4.9769913e-05
12,136 Automated Performance Management for the Big Data Stack 2019 CIDR 4.9769913e-05
12,169 Cost-Effective, Workload-Adaptive Migration of Big Data Applications to the Cloud 2019 SIGMOD 4.9769913e-05
12,195 An Experimental Evaluation of Garbage Collectors on Big Data Applications 2019 VLDB 4.9769913e-05
12,554 Workload Management for Big Data Analytics 2013 SIGMOD 4.9769913e-05
12,595 Optimization Strategies for A/B Testing on HADOOP 2013 VLDB 4.9769913e-05
Previous Page 1 / 1 Next

Outgoing Citations (Sorted by Pagerank)

Showing 8 of 8 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank Cited Paper Year Venue Pagerank
120 HadoopDB: An Architectural Hybrid of MapReduce and DBMS Technologies for Analytical Workloads 2009 VLDB 0.00031099083
384 HaLoop: Efficient Iterative Data Processing on Large Clusters 2010 VLDB 0.00019471648
675 Hadoop++: Making a Yellow Elephant Run Like a Cheetah (Without It Even Noticing) 2010 VLDB 0.00014880686
799 A Comparison of Join Algorithms for Log Processing in MapReduce 2010 SIGMOD 0.00013882649
823 MRShare: Sharing Across Multiple Queries in MapReduce 2010 VLDB 0.00013642066
940 Starfish: A Self-tuning System for Big Data Analytics 2011 CIDR 0.00012959992
1,282 Automatic Optimization for MapReduce Programs 2011 VLDB 0.00011208192
1,466 The Performance of MapReduce: An In-depth Study 2010 VLDB 0.00010565007
Previous Page 1 / 1 Next

Semantically Similar Papers

# Overall Rank Paper Year Venue
1 120 HadoopDB: An Architectural Hybrid of MapReduce and DBMS Technologies for Analytical Workloads 2009 VLDB
2 75 Map-Reduce-Merge: Simplified Relational Data Processing on Large Clusters 2007 SIGMOD
3 9,699 Efficient Big Data Processing in Hadoop MapReduce 2012 VLDB
4 1,073 Interactive Analytical Processing in Big Data Systems: A Cross-Industry Study of MapReduce Workloads 2012 VLDB
5 2,573 Minimal MapReduce Algorithms 2013 SIGMOD
6 1,466 The Performance of MapReduce: An In-depth Study 2010 VLDB
7 2,285 A Platform for Scalable One-Pass Analytics using MapReduce 2011 SIGMOD
8 3,217 Multi-Query Optimization in MapReduce Framework 2014 VLDB
9 1,282 Automatic Optimization for MapReduce Programs 2011 VLDB
10 8,671 MapReduce Programming and Cost-based Optimization? Crossing this Chasm with Starfish 2011 VLDB