DBScholar

Back to papers

Profiling, What-if Analysis, and Cost-based Optimization of MapReduce Programs

Summary: First cost-based optimizer for MapReduce programs, tackling the large configuration parameter space with black-box map/reduce functions. Profiler for unmodified programs and a what-if cost estimator enable data-driven optimization; prototype on Hadoop, with cross-domain evaluation. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID
10365
Venue
VLDB
Year
2011
Pagerank
0.00014522606
Overall Rank
735 | 94.96%
DOI
10.14778/3402703.3402707

Incoming Non-self Citations Over Time

Authors

BibTeX Citation

@article{herodotou_vldb11,
        title = {{Profiling, What-if Analysis, and Cost-based Optimization of MapReduce Programs}},
        author = {Herodotou, Herodotos and Babu, Shivnath},
        journal = {PVLDB},
        series = {{VLDB} '11},
        volume = {4},
        number = {11},
        pages = {1111--1122},
        doi = {10.14778/3402703.3402707},
        url = {https://doi.org/10.14778/3402703.3402707},
        year = {2011}
}

Incoming Citations (Sorted by Pagerank)

Showing 37 of 37 citing papers.

Rank Citing Paper Year Venue Pagerank
682 Robust Estimation of Resource Consumption for SQL Queries using Statistical Techniques 2012 VLDB 0.00015014887
1,054 Interactive Analytical Processing in Big Data Systems: A Cross-Industry Study of MapReduce Workloads 2012 VLDB 0.00012390673
1,079 Hybrid Parallelization Strategies for Large-Scale Machine Learning in SystemML 2014 VLDB 0.00012258469
1,739 Split Query Processing in Polybase 2013 SIGMOD 9.8814935e-05
1,983 Pregelix: Big(ger) Graph Analytics on A Dataflow Engine 2015 VLDB 9.3544951e-05
2,164 Opening the Black Boxes in Data Flow Optimization 2012 VLDB 9.0521951e-05
2,217 Cumulon: Optimizing Statistical Data Analysis in the Cloud 2013 SIGMOD 8.9332438e-05
2,486 Stubby: A Transformation-based Optimizer for MapReduce Workflows 2012 VLDB 8.5143189e-05
2,539 Minimal MapReduce Algorithms 2013 SIGMOD 8.4526595e-05
3,163 Multi-Query Optimization in MapReduce Framework 2014 VLDB 7.6784171e-05
3,494 WANalytics: Analytics for a Geo-Distributed Data-Intensive World 2015 CIDR 7.3642231e-05
4,049 Resource Elasticity for Large-Scale Machine Learning 2015 SIGMOD 6.9369379e-05
4,208 Clash of the Titans: MapReduce vs. Spark for Large Scale Data Analytics 2015 VLDB 6.8319812e-05
4,398 Real-time Workload Pattern Analysis for Large-scale Cloud Databases 2023 VLDB 6.7248611e-05
5,025 HYPER: Hypothetical Reasoning With What-If and How-To Queries Using a Probabilistic Causal Approach 2022 SIGMOD 6.3974766e-05
5,156 Only Aggressive Elephants are Fast Elephants 2012 VLDB 6.3410921e-05
5,287 PREDIcT: Towards Predicting the Runtime of Large Scale Iterative Analytics 2013 VLDB 6.2815429e-05
5,865 Speedup Your Analytics: Automatic Parameter Tuning for Databases and Big Data Systems 2019 VLDB 6.0619534e-05
6,423 Memory-Aware Framework for Efficient Second-Order Random Walk on Large Graphs 2020 SIGMOD 5.8813146e-05
6,576 Toward Interpretable and Actionable Data Analysis with Explanations and Causality 2022 VLDB 5.8364729e-05
6,736 Hadoop's Adolescence: An analysis of Hadoop usage in scientific workloads 2013 VLDB 5.787547e-05
6,946 KEA: Tuning an Exabyte-Scale Data Infrastructure 2021 SIGMOD 5.7309848e-05
7,111 Submodularity of Distributed Join Computation 2018 SIGMOD 5.69924e-05
7,242 Fast Multi-Column Sorting in Main-Memory Column-Stores 2016 SIGMOD 5.6641994e-05
7,546 MRTuner: A Toolkit to Enable Holistic Optimization for MapReduce Jobs 2014 VLDB 5.6024593e-05
8,497 MapReduce Programming and Cost-based Optimization? Crossing this Chasm with Starfish 2011 VLDB 5.41357e-05
8,927 Hyperspace: The Indexing Subsystem of Azure Synapse 2021 VLDB 5.3483178e-05
9,079 QMapper for Smart Grid: Migrating SQL-based Application to Hive 2015 SIGMOD 5.3251649e-05
9,510 Efficient Big Data Processing in Hadoop MapReduce 2012 VLDB 5.2576928e-05
9,640 Supporting Scalable Analytics with Latency Constraints 2015 VLDB 5.2434488e-05
9,902 UDAO: A Next-Generation Unified Data Analytics Optimizer 2019 VLDB 5.1997534e-05
11,566 PACk: An Efficient Partition-based Distributed Agglomerative Hierarchical Clustering Algorithm for Deduplication 2022 VLDB 5.093636e-05
11,829 Automated Performance Management for the Big Data Stack 2019 CIDR 5.093636e-05
11,863 Cost-Effective, Workload-Adaptive Migration of Big Data Applications to the Cloud 2019 SIGMOD 5.093636e-05
11,889 An Experimental Evaluation of Garbage Collectors on Big Data Applications 2019 VLDB 5.093636e-05
12,257 Workload Management for Big Data Analytics 2013 SIGMOD 5.093636e-05
12,298 Optimization Strategies for A/B Testing on HADOOP 2013 VLDB 5.093636e-05
Previous Page 1 / 1 Next

Outgoing Citations (Sorted by Pagerank)

Showing 8 of 8 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank Cited Paper Year Venue Pagerank
120 HadoopDB: An Architectural Hybrid of MapReduce and DBMS Technologies for Analytical Workloads 2009 VLDB 0.00031680027
372 HaLoop: Efficient Iterative Data Processing on Large Clusters 2010 VLDB 0.0001981521
660 Hadoop++: Making a Yellow Elephant Run Like a Cheetah (Without It Even Noticing) 2010 VLDB 0.00015198804
769 A Comparison of Join Algorithms for Log Processing in MapReduce 2010 SIGMOD 0.00014166872
803 MRShare: Sharing Across Multiple Queries in MapReduce 2010 VLDB 0.00013899943
923 Starfish: A Self-tuning System for Big Data Analytics 2011 CIDR 0.00013189886
1,257 Automatic Optimization for MapReduce Programs 2011 VLDB 0.000114432
1,436 The Performance of MapReduce: An In-depth Study 2010 VLDB 0.00010797443
Previous Page 1 / 1 Next

Semantically Similar Papers

# Overall Rank Paper Year Venue
1 120 HadoopDB: An Architectural Hybrid of MapReduce and DBMS Technologies for Analytical Workloads 2009 VLDB
2 72 Map-Reduce-Merge: Simplified Relational Data Processing on Large Clusters 2007 SIGMOD
3 9,510 Efficient Big Data Processing in Hadoop MapReduce 2012 VLDB
4 1,054 Interactive Analytical Processing in Big Data Systems: A Cross-Industry Study of MapReduce Workloads 2012 VLDB
5 2,539 Minimal MapReduce Algorithms 2013 SIGMOD
6 2,265 A Platform for Scalable One-Pass Analytics using MapReduce 2011 SIGMOD
7 1,436 The Performance of MapReduce: An In-depth Study 2010 VLDB
8 3,163 Multi-Query Optimization in MapReduce Framework 2014 VLDB
9 1,257 Automatic Optimization for MapReduce Programs 2011 VLDB
10 8,497 MapReduce Programming and Cost-based Optimization? Crossing this Chasm with Starfish 2011 VLDB