Rockhopper: A Robust Optimizer for Spark Configuration Tuning in Production Environment

Authors

• 1. Yiwen Zhu (Microsoft)
• 2. Rathijit Sen (Microsoft)
• 3. Brian P Kroth (Microsoft)
• 4. Sergiy Matusevych (Microsoft)
• 5. Andreas Mueller (Microsoft)
• 6. Tengfei Huang (Microsoft)
• 7. Rahul Challapalli (Microsoft)
• 8. Weihan Tang (Microsoft)
• 9. Estera Kot (Clouds on Mars)
• 10. Sule Kahraman (Microsoft)
• 11. Xin He (Microsoft)
• 12. Mo Liu (Microsoft)
• 13. Arshdeep Sekhon (Microsoft)
• 14. Dario Bernal (Microsoft)
• 15. Aditya Lakra (Microsoft)
• 16. Shaily Fozdar (Dynamo AI)
• 17. Dhruv Relwani (Microsoft)
• 18. Rui Fang (Microsoft)
• 19. Long Tian (Microsoft)
• 20. Karuna Sagar Krishna (Microsoft)
• 21. Ashit Gosalia (Microsoft)
• 22. Carlo Curino (Microsoft)
• 23. Subru Krishnan (Microsoft)

@inproceedings{zhu_sigmod25,
        title = {{Rockhopper: A Robust Optimizer for Spark Configuration Tuning in Production Environment}},
        author = {Zhu, Yiwen and Sen, Rathijit and Kroth, Brian P and Matusevych, Sergiy and Mueller, Andreas and Huang, Tengfei and Challapalli, Rahul and Tang, Weihan and Kot, Estera and Kahraman, Sule and He, Xin and Liu, Mo and Sekhon, Arshdeep and Bernal, Dario and Lakra, Aditya and Fozdar, Shaily and Relwani, Dhruv and Fang, Rui and Tian, Long and Krishna, Karuna Sagar and Gosalia, Ashit and Curino, Carlo and Krishnan, Subru},
        series = {{SIGMOD} '25},
        booktitle = {Proceedings of the {ACM} {SIGMOD} International Conference on Management of Data},
        publisher = {Association for Computing Machinery},
        doi = {10.1145/3722212.3724451},
        url = {https://dl.acm.org/doi/10.1145/3722212.3724451},
        year = {2025}
}

Incoming Citations (Sorted by Pagerank)

Showing 1 of 1 citing papers.

Outgoing Citations (Sorted by Pagerank)

Showing 17 of 17 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Semantically Similar Papers