DBScholar

Back to papers

Fine-Grained Modeling and Optimization for Intelligent Resource Management in Big Data Processing

Summary: Fine-grained instance-level modeling with a MaxCompute-based architecture decomposes resource optimization into simpler, multi-objective decisions (partition count, placement, per-instance resources). Novel predictive models and optimization methods enable sub-second RO and yield 37–72% latency and 43–78% cost reductions on production workloads. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID
h587cc64bbf3f0604
Venue
VLDB
Year
2022
Pagerank
6.2926774e-05
Overall Rank
5,058 | 66.00%
DOI
10.14778/3551793.3551855

Incoming Non-self Citations Over Time

Authors

BibTeX Citation

@article{lyu_vldb22,
        title = {{Fine-Grained Modeling and Optimization for Intelligent Resource Management in Big Data Processing}},
        author = {Lyu, Chenghao and Fan, Qi and Song, Fei and Sinha, Arnab and Diao, Yanlei and Chen, Wei and Ma, Li and Feng, Yihui and Li, Yaliang and Zeng, Kai and Zhou, Jingren},
        journal = {PVLDB},
        series = {{VLDB} '22},
        volume = {15},
        number = {11},
        pages = {3098--3111},
        doi = {10.14778/3551793.3551855},
        url = {https://doi.org/10.14778/3551793.3551855},
        year = {2022}
}

Incoming Citations (Sorted by Pagerank)

Showing 9 of 9 citing papers.

Rank Citing Paper Year Venue Pagerank
4,174 Resource Management in Aurora Serverless 2024 VLDB 6.7584856e-05
5,214 Stage: Query Execution Time Prediction in Amazon Redshift 2024 SIGMOD 6.2248104e-05
7,160 Cost-Intelligent Data Analytics in the Cloud 2024 CIDR 5.5962258e-05
7,460 T3: Accurate and Fast Performance Prediction for Relational Database Systems With Compiled Decision Trees 2025 SIGMOD 5.5215755e-05
7,977 The Holon Approach for Simultaneously Tuning Multiple Components in a Self-Driving Database Management System with Machine Learning via Synthesized Proto-Actions 2024 VLDB 5.4142519e-05
8,004 A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning 2024 VLDB 5.4089097e-05
9,956 Graph Transformers for Query Plan Representation: Potentials and Challenges 2025 VLDB 5.1038322e-05
10,148 Improving DBMS Scheduling Decisions with Accurate Performance Prediction on Concurrent Queries 2025 VLDB 5.0715586e-05
10,943 Lakebase: Serverless Postgres over Open Lake Storage 2026 VLDB 4.9793485e-05
Previous Page 1 / 1 Next

Outgoing Citations (Sorted by Pagerank)

Showing 41 of 41 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank Cited Paper Year Venue Pagerank
31 Hive - A Warehousing Solution Over a Map-Reduce Framework 2009 VLDB 0.00049839909
78 Automatic Database Management System Tuning Through Large-scale Machine Learning 2017 SIGMOD 0.00036684414
145 Neo: A Learned Query Optimizer 2019 VLDB 0.0002908188
314 An End-to-End Automatic Cloud Database Tuning System Using Deep Reinforcement Learning 2019 SIGMOD 0.00021282642
406 Deep Unsupervised Cardinality Estimation 2020 VLDB 0.00019045544
432 Shark: SQL and Rich Analytics at Scale 2013 SIGMOD 0.00018339357
461 An End-to-End Learning-based Cost Estimator 2020 VLDB 0.00017829982
512 NeuroCard: One Cardinality Estimator for All Tables 2021 VLDB 0.00017050173
560 Plan-Structured Deep Neural Network Models for Query Performance Prediction 2019 VLDB 0.00016403151
651 Building a High-Level Dataflow System on top of Map-Reduce: The Pig Experience 2009 VLDB 0.00015130782
1,298 Parametric Query Optimization for Linear and Piecewise Linear Cost Functions 2002 VLDB 0.00011120288
1,433 Towards a Learning Optimizer for Shared Clouds 2019 VLDB 0.00010677711
1,580 Deep Learning Models for Selectivity Estimation of Multi-Attribute Queries 2020 SIGMOD 0.00010180835
1,734 Flow-Loss: Learning Cardinality Estimates That Matter 2021 VLDB 9.7545773e-05
2,004 FLAT: Fast, Lightweight and Accurate Method for Cardinality Estimation 2021 VLDB 9.2065719e-05
2,213 Towards Cost-Optimal Query Processing in the Cloud 2021 VLDB 8.8206114e-05
2,228 Fuxi: a Fault-Tolerant Resource Management and Job Scheduling System at Internet Scale 2014 VLDB 8.8020387e-05
2,524 Multi-Objective Parametric Query Optimization 2015 VLDB 8.3439125e-05
2,583 Fauce: Fast and Accurate Deep Ensembles with Uncertainty for Cardinality Estimation 2021 VLDB 8.2589758e-05
2,720 ResTune: Resource Oriented Tuning Boosted by Meta-Learning for Cloud Databases 2021 SIGMOD 8.0966919e-05
2,834 Cost Models for Big Data Query Processing: Learning, Retrofitting, and Our Findings 2020 SIGMOD 7.9560627e-05
2,842 Query Performance Prediction for Concurrent Queries using Graph Embedding 2020 VLDB 7.949193e-05
2,981 WiSeDB: A Learning-based Workload Management Advisor for Cloud Databases 2016 VLDB 7.7851845e-05
3,052 A Unified Deep Model of Learning from both Data and Queries for Cardinality Estimation 2021 SIGMOD 7.7052471e-05
3,741 FACE: A Normalizing Flow based Cardinality Estimator 2022 VLDB 7.0594076e-05
3,965 MB2: Decomposed Behavior Modeling for Self-Driving Database Management Systems 2021 SIGMOD 6.8918628e-05
4,188 Schedule Optimization for Data Processing Flows on the Cloud 2011 SIGMOD 6.7464862e-05
4,688 Learned Cardinality Estimation for Similarity Queries 2021 SIGMOD 6.4697463e-05
4,980 An Incremental Anytime Algorithm for Multi-Objective Query Optimization 2015 SIGMOD 6.3293674e-05
5,039 Leveraging Query Logs and Machine Learning for Parametric Query Optimization 2022 VLDB 6.3023214e-05
5,223 Approximation Schemes for Many-Objective Query Optimization 2014 SIGMOD 6.2212838e-05
5,840 Pre-training Summarization Models of Structured Datasets for Cardinality Estimation 2022 VLDB 5.9737703e-05
7,315 Model-Free Control for Distributed Stream Data Processing using Deep Reinforcement Learning 2018 VLDB 5.554799e-05
7,364 Weighted Distinct Sampling: Cardinality Estimation for SPJ Queries 2021 SIGMOD 5.5418075e-05
7,967 Resource Bricolage for Parallel Database Systems 2015 VLDB 5.415894e-05
8,650 PostCENN: PostgreSQL with Machine Learning Models for Cardinality Estimation 2021 VLDB 5.2948655e-05
9,190 Tempo: Robust and Self-Tuning Resource Management in Multi-tenant Parallel Databases 2016 VLDB 5.2112843e-05
9,358 Phoebe: A Learning-based Checkpoint Optimizer 2021 VLDB 5.1869771e-05
9,823 Trident: Task Scheduling over Tiered Storage Systems in Big Data Platforms 2021 VLDB 5.1254334e-05
9,824 Optimistic Recovery for Iterative Dataflows in Action 2015 SIGMOD 5.1254334e-05
10,088 UDAO: A Next-Generation Unified Data Analytics Optimizer 2019 VLDB 5.0830849e-05
Previous Page 1 / 1 Next

Semantically Similar Papers

# Overall Rank Paper Year Venue
1 2,305 Multi-dimensional Resource Scheduling for Parallel Queries 1996 SIGMOD
2 3,440 Advanced Partitioning Techniques for Massively Distributed Computation 2012 SIGMOD
3 4,116 Resource Elasticity for Large-Scale Machine Learning 2015 SIGMOD
4 3,216 Multi-Query Optimization in MapReduce Framework 2014 VLDB
5 4,188 Schedule Optimization for Data Processing Flows on the Cloud 2011 SIGMOD
6 4,818 Continuous Cloud-Scale Query Optimization and Processing 2013 VLDB
7 8,004 A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning 2024 VLDB
8 2,834 Cost Models for Big Data Query Processing: Learning, Retrofitting, and Our Findings 2020 SIGMOD
9 11,921 Budget-Conscious Fine-Grained Configuration Optimization for Spatio-Temporal Applications 2022 VLDB
10 9,816 Supporting Scalable Analytics with Latency Constraints 2015 VLDB