Database Paper Browser

Back to papers

Fine-Grained Modeling and Optimization for Intelligent Resource Management in Big Data Processing

Summary: Fine-grained instance-level modeling with a MaxCompute-based architecture decomposes resource optimization into simpler, multi-objective decisions (partition count, placement, per-instance resources). Novel predictive models and optimization methods enable sub-second RO and yield 37–72% latency and 43–78% cost reductions on production workloads. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID
12791
Venue
VLDB
Year
2022
Pagerank
5.5410059e-05
Overall Rank
5,373 | 62.66%
DOI
10.14778/3551793.3551855

Incoming Non-self Citations Over Time

Authors

Incoming Citations (Sorted by Pagerank)

Showing 8 of 8 citing papers.

Rank Citing Paper Year Venue Pagerank
5,712 Resource Management in Aurora Serverless 2024 VLDB 5.3596268e-05
5,844 Stage: Query Execution Time Prediction in Amazon Redshift 2024 SIGMOD 5.3060581e-05
7,787 Cost-Intelligent Data Analytics in the Cloud 2024 CIDR 4.6468106e-05
8,003 The Holon Approach for Simultaneously Tuning Multiple Components in a Self-Driving Database Management System with Machine Learning via Synthesized Proto-Actions 2024 VLDB 4.6049527e-05
8,585 A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning 2024 VLDB 4.4856045e-05
8,961 T3: Accurate and Fast Performance Prediction for Relational Database Systems With Compiled Decision Trees 2025 SIGMOD 4.4171776e-05
10,733 Improving DBMS Scheduling Decisions with Accurate Performance Prediction on Concurrent Queries 2025 VLDB 4.1905499e-05
10,863 Graph Transformers for Query Plan Representation: Potentials and Challenges 2025 VLDB 4.1905499e-05
Previous Page 1 / 1 Next

Outgoing Citations (Sorted by Pagerank)

Showing 41 of 41 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank Cited Paper Year Venue Pagerank
70 Hive - A Warehousing Solution Over a Map-Reduce Framework 2009 VLDB 0.00059744625
183 Automatic Database Management System Tuning Through Large-scale Machine Learning 2017 SIGMOD 0.00036859633
329 Neo: A Learned Query Optimizer 2019 VLDB 0.00027301488
510 An End-to-End Automatic Cloud Database Tuning System Using Deep Reinforcement Learning 2019 SIGMOD 0.00021420477
539 Shark: SQL and Rich Analytics at Scale 2013 SIGMOD 0.00020615453
752 Deep Unsupervised Cardinality Estimation 2020 VLDB 0.00017138049
776 Building a High-Level Dataflow System on top of Map-Reduce: The Pig Experience 2009 VLDB 0.00016765694
804 An End-to-End Learning-based Cost Estimator 2020 VLDB 0.0001643674
876 Plan-Structured Deep Neural Network Models for Query Performance Prediction 2019 VLDB 0.00015660534
905 NeuroCard: One Cardinality Estimator for All Tables 2021 VLDB 0.00015423174
1,644 Parametric Query Optimization for Linear and Piecewise Linear Cost Functions 2002 VLDB 0.0001102889
1,678 Fuxi: a Fault-Tolerant Resource Management and Job Scheduling System at Internet Scale 2014 VLDB 0.00010928011
2,080 Towards a Learning Optimizer for Shared Clouds 2019 VLDB 9.5954034e-05
2,364 Deep Learning Models for Selectivity Estimation of Multi-Attribute Queries 2020 SIGMOD 8.955077e-05
2,553 Towards Cost-Optimal Query Processing in the Cloud 2021 VLDB 8.5522099e-05
2,652 Multi-Objective Parametric Query Optimization 2015 VLDB 8.3662031e-05
2,769 FLAT: Fast, Lightweight and Accurate Method for Cardinality Estimation 2021 VLDB 8.1512848e-05
2,781 Flow-Loss: Learning Cardinality Estimates That Matter 2021 VLDB 8.1282042e-05
3,222 WiSeDB: A Learning-based Workload Management Advisor for Cloud Databases 2016 VLDB 7.3531422e-05
3,492 Fauce: Fast and Accurate Deep Ensembles with Uncertainty for Cardinality Estimation 2021 VLDB 7.0435484e-05
3,580 Query Performance Prediction for Concurrent Queries using Graph Embedding 2020 VLDB 6.9460425e-05
3,623 Cost Models for Big Data Query Processing: Learning, Retrofitting, and Our Findings 2020 SIGMOD 6.9017341e-05
3,924 A Unified Deep Model of Learning from both Data and Queries for Cardinality Estimation 2021 SIGMOD 6.6227223e-05
3,995 ResTune: Resource Oriented Tuning Boosted by Meta-Learning for Cloud Databases 2021 SIGMOD 6.5475871e-05
4,543 FACE: A Normalizing Flow based Cardinality Estimator 2022 VLDB 6.0953507e-05
4,587 MB2: Decomposed Behavior Modeling for Self-Driving Database Management Systems 2021 SIGMOD 6.0594195e-05
4,698 Schedule Optimization for Data Processing Flows on the Cloud 2011 SIGMOD 5.9835195e-05
4,876 Approximation Schemes for Many-Objective Query Optimization 2014 SIGMOD 5.8544467e-05
5,073 An Incremental Anytime Algorithm for Multi-Objective Query Optimization 2015 SIGMOD 5.7118738e-05
5,477 Learned Cardinality Estimation for Similarity Queries 2021 SIGMOD 5.4856699e-05
6,365 Pre-training Summarization Models of Structured Datasets for Cardinality Estimation 2022 VLDB 5.0892829e-05
6,639 Leveraging Query Logs and Machine Learning for Parametric Query Optimization 2022 VLDB 4.976781e-05
7,340 Weighted Distinct Sampling: Cardinality Estimation for SPJ Queries 2021 SIGMOD 4.7526052e-05
7,366 Model-Free Control for Distributed Stream Data Processing using Deep Reinforcement Learning 2018 VLDB 4.7459576e-05
7,914 Resource Bricolage for Parallel Database Systems 2015 VLDB 4.6136905e-05
8,573 PostCENN: PostgreSQL with Machine Learning Models for Cardinality Estimation 2021 VLDB 4.4885745e-05
9,064 Tempo: Robust and Self-Tuning Resource Management in Multi-tenant Parallel Databases 2016 VLDB 4.3994049e-05
9,138 Phoebe: A Learning-based Checkpoint Optimizer 2021 VLDB 4.3842765e-05
9,547 Trident: Task Scheduling over Tiered Storage Systems in Big Data Platforms 2021 VLDB 4.3219254e-05
9,548 Optimistic Recovery for Iterative Dataflows in Action 2015 SIGMOD 4.3219254e-05
9,735 UDAO: A Next-Generation Unified Data Analytics Optimizer 2019 VLDB 4.2901665e-05
Previous Page 1 / 1 Next

Semantically Similar Papers

Overall Rank Paper Year Venue Pagerank
2,458 Multi-dimensional Resource Scheduling for Parallel Queries 1996 SIGMOD 8.7601979e-05
4,068 Advanced Partitioning Techniques for Massively Distributed Computation 2012 SIGMOD 6.4748133e-05
4,807 Resource Elasticity for Large-Scale Machine Learning 2015 SIGMOD 5.9045148e-05
3,709 Multi-Query Optimization in MapReduce Framework 2014 VLDB 6.8211506e-05
4,698 Schedule Optimization for Data Processing Flows on the Cloud 2011 SIGMOD 5.9835195e-05
5,309 Continuous Cloud-Scale Query Optimization and Processing 2013 VLDB 5.5714729e-05
3,623 Cost Models for Big Data Query Processing: Learning, Retrofitting, and Our Findings 2020 SIGMOD 6.9017341e-05
8,585 A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning 2024 VLDB 4.4856045e-05
11,418 Budget-Conscious Fine-Grained Configuration Optimization for Spatio-Temporal Applications 2022 VLDB 4.1905499e-05
9,505 Supporting Scalable Analytics with Latency Constraints 2015 VLDB 4.3300131e-05