DBScholar

Back to papers

Fine-Grained Modeling and Optimization for Intelligent Resource Management in Big Data Processing

Summary: Fine-grained instance-level modeling with a MaxCompute-based architecture decomposes resource optimization into simpler, multi-objective decisions (partition count, placement, per-instance resources). Novel predictive models and optimization methods enable sub-second RO and yield 37–72% latency and 43–78% cost reductions on production workloads. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID
12978
Venue
VLDB
Year
2022
Pagerank
6.2362811e-05
Overall Rank
5,388 | 63.04%
DOI
10.14778/3551793.3551855

Incoming Non-self Citations Over Time

Authors

BibTeX Citation

@article{lyu_vldb22,
        title = {{Fine-Grained Modeling and Optimization for Intelligent Resource Management in Big Data Processing}},
        author = {Lyu, Chenghao and Fan, Qi and Song, Fei and Sinha, Arnab and Diao, Yanlei and Chen, Wei and Ma, Li and Feng, Yihui and Li, Yaliang and Zeng, Kai and Zhou, Jingren},
        journal = {PVLDB},
        series = {{VLDB} '22},
        volume = {15},
        number = {11},
        pages = {3098--3111},
        doi = {10.14778/3551793.3551855},
        url = {https://doi.org/10.14778/3551793.3551855},
        year = {2022}
}

Incoming Citations (Sorted by Pagerank)

Showing 8 of 8 citing papers.

Rank Citing Paper Year Venue Pagerank
5,107 Stage: Query Execution Time Prediction in Amazon Redshift 2024 SIGMOD 6.3623786e-05
5,188 Resource Management in Aurora Serverless 2024 VLDB 6.3278921e-05
7,589 Cost-Intelligent Data Analytics in the Cloud 2024 CIDR 5.5907048e-05
7,846 The Holon Approach for Simultaneously Tuning Multiple Components in a Self-Driving Database Management System with Machine Learning via Synthesized Proto-Actions 2024 VLDB 5.5331459e-05
8,572 T3: Accurate and Fast Performance Prediction for Relational Database Systems With Compiled Decision Trees 2025 SIGMOD 5.4102362e-05
8,615 A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning 2024 VLDB 5.4005602e-05
10,969 Improving DBMS Scheduling Decisions with Accurate Performance Prediction on Concurrent Queries 2025 VLDB 5.093636e-05
11,083 Graph Transformers for Query Plan Representation: Potentials and Challenges 2025 VLDB 5.093636e-05
Previous Page 1 / 1 Next

Outgoing Citations (Sorted by Pagerank)

Showing 41 of 41 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank Cited Paper Year Venue Pagerank
32 Hive - A Warehousing Solution Over a Map-Reduce Framework 2009 VLDB 0.00050111008
86 Automatic Database Management System Tuning Through Large-scale Machine Learning 2017 SIGMOD 0.00035316107
154 Neo: A Learned Query Optimizer 2019 VLDB 0.00028726181
334 An End-to-End Automatic Cloud Database Tuning System Using Deep Reinforcement Learning 2019 SIGMOD 0.00020875082
401 Deep Unsupervised Cardinality Estimation 2020 VLDB 0.00019092557
425 Shark: SQL and Rich Analytics at Scale 2013 SIGMOD 0.00018704491
465 An End-to-End Learning-based Cost Estimator 2020 VLDB 0.0001803934
513 NeuroCard: One Cardinality Estimator for All Tables 2021 VLDB 0.00017190574
563 Plan-Structured Deep Neural Network Models for Query Performance Prediction 2019 VLDB 0.0001650812
642 Building a High-Level Dataflow System on top of Map-Reduce: The Pig Experience 2009 VLDB 0.00015395331
1,321 Parametric Query Optimization for Linear and Piecewise Linear Cost Functions 2002 VLDB 0.00011162369
1,468 Towards a Learning Optimizer for Shared Clouds 2019 VLDB 0.00010686496
1,573 Deep Learning Models for Selectivity Estimation of Multi-Attribute Queries 2020 SIGMOD 0.00010328171
1,876 Flow-Loss: Learning Cardinality Estimates That Matter 2021 VLDB 9.5717543e-05
1,988 FLAT: Fast, Lightweight and Accurate Method for Cardinality Estimation 2021 VLDB 9.3501502e-05
2,331 Fuxi: a Fault-Tolerant Resource Management and Job Scheduling System at Internet Scale 2014 VLDB 8.7430753e-05
2,344 Towards Cost-Optimal Query Processing in the Cloud 2021 VLDB 8.7199754e-05
2,540 Multi-Objective Parametric Query Optimization 2015 VLDB 8.45187e-05
2,620 Fauce: Fast and Accurate Deep Ensembles with Uncertainty for Cardinality Estimation 2021 VLDB 8.3363963e-05
2,812 Query Performance Prediction for Concurrent Queries using Graph Embedding 2020 VLDB 8.0979597e-05
2,822 Cost Models for Big Data Query Processing: Learning, Retrofitting, and Our Findings 2020 SIGMOD 8.0898536e-05
2,958 WiSeDB: A Learning-based Workload Management Advisor for Cloud Databases 2016 VLDB 7.9197796e-05
3,086 A Unified Deep Model of Learning from both Data and Queries for Cardinality Estimation 2021 SIGMOD 7.7708642e-05
3,116 ResTune: Resource Oriented Tuning Boosted by Meta-Learning for Cloud Databases 2021 SIGMOD 7.7390737e-05
3,688 FACE: A Normalizing Flow based Cardinality Estimator 2022 VLDB 7.201795e-05
3,961 MB2: Decomposed Behavior Modeling for Self-Driving Database Management Systems 2021 SIGMOD 6.987575e-05
4,117 Schedule Optimization for Data Processing Flows on the Cloud 2011 SIGMOD 6.8919486e-05
4,617 Learned Cardinality Estimation for Similarity Queries 2021 SIGMOD 6.604437e-05
4,880 An Incremental Anytime Algorithm for Multi-Objective Query Optimization 2015 SIGMOD 6.4656221e-05
5,124 Approximation Schemes for Many-Objective Query Optimization 2014 SIGMOD 6.3567981e-05
5,394 Leveraging Query Logs and Machine Learning for Parametric Query Optimization 2022 VLDB 6.2336084e-05
5,792 Pre-training Summarization Models of Structured Datasets for Cardinality Estimation 2022 VLDB 6.0871213e-05
7,194 Model-Free Control for Distributed Stream Data Processing using Deep Reinforcement Learning 2018 VLDB 5.6766532e-05
7,256 Weighted Distinct Sampling: Cardinality Estimation for SPJ Queries 2021 SIGMOD 5.6625146e-05
7,822 Resource Bricolage for Parallel Database Systems 2015 VLDB 5.5371287e-05
8,581 PostCENN: PostgreSQL with Machine Learning Models for Cardinality Estimation 2021 VLDB 5.4082749e-05
9,039 Tempo: Robust and Self-Tuning Resource Management in Multi-tenant Parallel Databases 2016 VLDB 5.3267474e-05
9,224 Phoebe: A Learning-based Checkpoint Optimizer 2021 VLDB 5.3035811e-05
9,669 Trident: Task Scheduling over Tiered Storage Systems in Big Data Platforms 2021 VLDB 5.2389261e-05
9,670 Optimistic Recovery for Iterative Dataflows in Action 2015 SIGMOD 5.2389261e-05
9,902 UDAO: A Next-Generation Unified Data Analytics Optimizer 2019 VLDB 5.1997534e-05
Previous Page 1 / 1 Next

Semantically Similar Papers

# Overall Rank Paper Year Venue
1 2,337 Multi-dimensional Resource Scheduling for Parallel Queries 1996 SIGMOD
2 3,466 Advanced Partitioning Techniques for Massively Distributed Computation 2012 SIGMOD
3 4,049 Resource Elasticity for Large-Scale Machine Learning 2015 SIGMOD
4 3,163 Multi-Query Optimization in MapReduce Framework 2014 VLDB
5 4,117 Schedule Optimization for Data Processing Flows on the Cloud 2011 SIGMOD
6 4,742 Continuous Cloud-Scale Query Optimization and Processing 2013 VLDB
7 2,822 Cost Models for Big Data Query Processing: Learning, Retrofitting, and Our Findings 2020 SIGMOD
8 8,615 A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning 2024 VLDB
9 11,613 Budget-Conscious Fine-Grained Configuration Optimization for Spatio-Temporal Applications 2022 VLDB
10 9,640 Supporting Scalable Analytics with Latency Constraints 2015 VLDB