DBScholar

Back to papers

Fine-Grained Modeling and Optimization for Intelligent Resource Management in Big Data Processing

Summary: Fine-grained instance-level modeling with a MaxCompute-based architecture decomposes resource optimization into simpler, multi-objective decisions (partition count, placement, per-instance resources). Novel predictive models and optimization methods enable sub-second RO and yield 37–72% latency and 43–78% cost reductions on production workloads. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID
h587cc64bbf3f0604
Venue
VLDB
Year
2022
Pagerank
6.28918e-05
Overall Rank
5,043 | 66.20%
DOI
10.14778/3551793.3551855
PDF
Download (CC BY-NC-ND 4.0)

Incoming Non-self Citations Over Time

Authors

BibTeX Citation

@article{lyu_vldb22,
        title = {{Fine-Grained Modeling and Optimization for Intelligent Resource Management in Big Data Processing}},
        author = {Lyu, Chenghao and Fan, Qi and Song, Fei and Sinha, Arnab and Diao, Yanlei and Chen, Wei and Ma, Li and Feng, Yihui and Li, Yaliang and Zeng, Kai and Zhou, Jingren},
        journal = {PVLDB},
        series = {{VLDB} '22},
        volume = {15},
        number = {11},
        pages = {3098--3111},
        doi = {10.14778/3551793.3551855},
        url = {https://doi.org/10.14778/3551793.3551855},
        year = {2022}
}

Incoming Citations (Sorted by Pagerank)

Showing 9 of 9 citing papers.

Rank Citing Paper Year Venue Pagerank
3,998 Resource Management in Aurora Serverless 2024 VLDB 6.8497732e-05
4,833 Stage: Query Execution Time Prediction in Amazon Redshift 2024 SIGMOD 6.3834856e-05
7,169 Cost-Intelligent Data Analytics in the Cloud 2024 CIDR 5.5789764e-05
7,483 T3: Accurate and Fast Performance Prediction for Relational Database Systems With Compiled Decision Trees 2025 SIGMOD 5.5007125e-05
7,997 The Holon Approach for Simultaneously Tuning Multiple Components in a Self-Driving Database Management System with Machine Learning via Synthesized Proto-Actions 2024 VLDB 5.3938336e-05
8,026 A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning 2024 VLDB 5.3884724e-05
9,011 Improving DBMS Scheduling Decisions with Accurate Performance Prediction on Concurrent Queries 2025 VLDB 5.2238303e-05
9,995 Graph Transformers for Query Plan Representation: Potentials and Challenges 2025 VLDB 5.0845476e-05
11,013 Lakebase: Serverless Postgres over Open Lake Storage 2026 VLDB 4.9605343e-05
Previous Page 1 / 1 Next

Outgoing Citations (Sorted by Pagerank)

Showing 41 of 41 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank Cited Paper Year Venue Pagerank
31 Hive - A Warehousing Solution Over a Map-Reduce Framework 2009 VLDB 0.00049707539
74 Automatic Database Management System Tuning Through Large-scale Machine Learning 2017 SIGMOD 0.00037164008
142 Neo: A Learned Query Optimizer 2019 VLDB 0.00029114937
309 An End-to-End Automatic Cloud Database Tuning System Using Deep Reinforcement Learning 2019 SIGMOD 0.0002143249
405 Deep Unsupervised Cardinality Estimation 2020 VLDB 0.00019020138
436 Shark: SQL and Rich Analytics at Scale 2013 SIGMOD 0.00018274105
453 An End-to-End Learning-based Cost Estimator 2020 VLDB 0.0001801482
508 NeuroCard: One Cardinality Estimator for All Tables 2021 VLDB 0.00017108973
545 Plan-Structured Deep Neural Network Models for Query Performance Prediction 2019 VLDB 0.0001657599
655 Building a High-Level Dataflow System on top of Map-Reduce: The Pig Experience 2009 VLDB 0.00015076669
1,304 Parametric Query Optimization for Linear and Piecewise Linear Cost Functions 2002 VLDB 0.00011088983
1,436 Towards a Learning Optimizer for Shared Clouds 2019 VLDB 0.00010658836
1,583 Deep Learning Models for Selectivity Estimation of Multi-Attribute Queries 2020 SIGMOD 0.00010148362
1,740 Flow-Loss: Learning Cardinality Estimates That Matter 2021 VLDB 9.7289523e-05
2,009 FLAT: Fast, Lightweight and Accurate Method for Cardinality Estimation 2021 VLDB 9.1813272e-05
2,162 Towards Cost-Optimal Query Processing in the Cloud 2021 VLDB 8.9192007e-05
2,181 Fuxi: a Fault-Tolerant Resource Management and Job Scheduling System at Internet Scale 2014 VLDB 8.9007414e-05
2,531 Multi-Objective Parametric Query Optimization 2015 VLDB 8.3140235e-05
2,572 ResTune: Resource Oriented Tuning Boosted by Meta-Learning for Cloud Databases 2021 SIGMOD 8.2581875e-05
2,584 Fauce: Fast and Accurate Deep Ensembles with Uncertainty for Cardinality Estimation 2021 VLDB 8.2435322e-05
2,768 Query Performance Prediction for Concurrent Queries using Graph Embedding 2020 VLDB 8.0199086e-05
2,828 Cost Models for Big Data Query Processing: Learning, Retrofitting, and Our Findings 2020 SIGMOD 7.9476509e-05
2,892 WiSeDB: A Learning-based Workload Management Advisor for Cloud Databases 2016 VLDB 7.8872299e-05
3,060 A Unified Deep Model of Learning from both Data and Queries for Cardinality Estimation 2021 SIGMOD 7.6828463e-05
3,740 FACE: A Normalizing Flow based Cardinality Estimator 2022 VLDB 7.0398782e-05
3,959 MB2: Decomposed Behavior Modeling for Self-Driving Database Management Systems 2021 SIGMOD 6.8804723e-05
4,209 Schedule Optimization for Data Processing Flows on the Cloud 2011 SIGMOD 6.7214185e-05
4,703 Learned Cardinality Estimation for Similarity Queries 2021 SIGMOD 6.4458272e-05
5,006 An Incremental Anytime Algorithm for Multi-Objective Query Optimization 2015 SIGMOD 6.3057396e-05
5,062 Leveraging Query Logs and Machine Learning for Parametric Query Optimization 2022 VLDB 6.2791511e-05
5,248 Approximation Schemes for Many-Objective Query Optimization 2014 SIGMOD 6.1980997e-05
5,837 Pre-training Summarization Models of Structured Datasets for Cardinality Estimation 2022 VLDB 5.9600736e-05
7,333 Model-Free Control for Distributed Stream Data Processing using Deep Reinforcement Learning 2018 VLDB 5.5344269e-05
7,387 Weighted Distinct Sampling: Cardinality Estimation for SPJ Queries 2021 SIGMOD 5.5212177e-05
7,987 Resource Bricolage for Parallel Database Systems 2015 VLDB 5.3957073e-05
8,675 PostCENN: PostgreSQL with Machine Learning Models for Cardinality Estimation 2021 VLDB 5.2751633e-05
9,219 Tempo: Robust and Self-Tuning Resource Management in Multi-tenant Parallel Databases 2016 VLDB 5.191841e-05
9,394 Phoebe: A Learning-based Checkpoint Optimizer 2021 VLDB 5.1676892e-05
9,863 Trident: Task Scheduling over Tiered Storage Systems in Big Data Platforms 2021 VLDB 5.1063146e-05
9,864 Optimistic Recovery for Iterative Dataflows in Action 2015 SIGMOD 5.1063146e-05
10,127 UDAO: A Next-Generation Unified Data Analytics Optimizer 2019 VLDB 5.0638787e-05
Previous Page 1 / 1 Next

Semantically Similar Papers

# Overall Rank Paper Year Venue
1 2,309 Multi-dimensional Resource Scheduling for Parallel Queries 1996 SIGMOD
2 4,134 Resource Elasticity for Large-Scale Machine Learning 2015 SIGMOD
3 3,453 Advanced Partitioning Techniques for Massively Distributed Computation 2012 SIGMOD
4 3,228 Multi-Query Optimization in MapReduce Framework 2014 VLDB
5 4,209 Schedule Optimization for Data Processing Flows on the Cloud 2011 SIGMOD
6 4,842 Continuous Cloud-Scale Query Optimization and Processing 2013 VLDB
7 8,026 A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning 2024 VLDB
8 2,828 Cost Models for Big Data Query Processing: Learning, Retrofitting, and Our Findings 2020 SIGMOD
9 11,971 Budget-Conscious Fine-Grained Configuration Optimization for Spatio-Temporal Applications 2022 VLDB
10 9,857 Supporting Scalable Analytics with Latency Constraints 2015 VLDB