Towards Resource Efficiency: Practical Insights into Large-Scale Spark Workloads at ByteDance

Summary: Governance for large-scale Spark: push-based Cloud Shuffle + ESS to cut I/O stalls, and extended configs (milliCores, memoryBurst, spill modes) for fine-grained resource control. Two-stage auto-tuning on millions of jobs yields +22% CPU, +5% memory, 10% less shuffle time. (summarized by gpt-5-mini on Feb 09 2026)

Paper ID: 13582
Venue: VLDB
Year: 2024
Pagerank: 5.5057547e-05
Overall Rank: 8,333 | 42.09%
DOI: 10.14778/3685800.3685804

Incoming Non-self Citations Over Time

Authors

Incoming Citations (Sorted by Pagerank)

Showing 3 of 3 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
9,628	Rockhopper: A Robust Optimizer for Spark Configuration Tuning in Production Environment	2025	SIGMOD	5.3018378e-05
10,278	LakeHelm: Zero-Shot Lakehouse Advisor for Joint Engine-Format Selection and Configuration	2026	VLDB	5.1725247e-05
10,783	Magnus: A Holistic Approach to Data Management for Large-Scale Machine Learning Workloads	2025	VLDB	5.1725247e-05

Outgoing Citations (Sorted by Pagerank)

Showing 7 of 7 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
3,161	ResTune: Resource Oriented Tuning Boosted by Meta-Learning for Cloud Databases	2021	SIGMOD	7.7501079e-05
3,168	iBTune: Individualized Buffer Tuning for Large-scale Cloud Databases	2019	VLDB	7.7448853e-05
3,238	Why You Should Run TPC-DS: A Workload Analysis	2007	VLDB	7.6790472e-05
4,781	LOCAT: Low-Overhead Online Configuration Auto-Tuning of Spark SQL Applications	2022	SIGMOD	6.5784829e-05
5,328	Magnet: Push-based Shuffle Service for Large-scale Data Processing	2020	VLDB	6.3311481e-05
6,014	AutoExecutor: Predictive Parallelism for Spark SQL Queries	2021	VLDB	6.0692734e-05
6,248	Towards General and Efficient Online Tuning for Spark	2023	VLDB	6.0022621e-05

Semantically Similar Papers

Overall Rank	Paper	Year	Venue	Pagerank
9,722	SparkCruise: Handsfree Computation Reuse in Spark	2019	VLDB	5.2849709e-05
5,729	StreamOps: Cloud-Native Runtime Management for Streaming Services in ByteDance	2023	VLDB	6.170083e-05
3,358	Scaling Spark in the Real World: Performance and Usability	2015	VLDB	7.5512783e-05
5,741	Speedup Your Analytics: Automatic Parameter Tuning for Databases and Big Data Systems	2019	VLDB	6.1656808e-05
11,081	ResLake: Towards Minimum Job Latency and Balanced Resource Utilization in Geo-distributed Job Scheduling	2024	VLDB	5.1725247e-05
8,046	SparkCruise: Workload Optimization in Managed Spark Clusters at Microsoft	2021	VLDB	5.5585663e-05
11,014	Intelligent Pooling: Proactive Resource Provisioning in Large-scale Cloud Service	2024	VLDB	5.1725247e-05
8,472	A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning	2024	VLDB	5.4888877e-05
5,328	Magnet: Push-based Shuffle Service for Large-scale Data Processing	2020	VLDB	6.3311481e-05
6,248	Towards General and Efficient Online Tuning for Spark	2023	VLDB	6.0022621e-05